Oct 29 22:14:26.821668 ip-10-0-133-251 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:14:27.178636 ip-10-0-133-251 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:27.178636 ip-10-0-133-251 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:14:27.178636 ip-10-0-133-251 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:27.178636 ip-10-0-133-251 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:14:27.185333 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185242 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:27.185333 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185328 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:27.185333 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185333 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:27.185333 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185337 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:27.185333 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185341 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185345 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185349 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185353 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185358 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185361 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185366 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185369 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185373 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185377 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185381 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185385 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185388 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185407 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185411 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185415 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185419 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185422 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185426 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185431 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:27.185677 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185434 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185438 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185442 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185446 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185450 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185454 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185458 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185469 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185473 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185477 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185481 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185488 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185496 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185501 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185507 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185513 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185517 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185522 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185526 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:27.186258 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185531 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185535 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185540 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185544 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185548 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185552 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185556 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185560 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185565 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185569 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185573 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185578 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185582 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185586 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185590 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185594 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185599 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185603 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185607 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185611 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:27.186793 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185616 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185620 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185626 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185631 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185635 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185640 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185644 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185649 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185654 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185658 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185662 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185666 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185670 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185674 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185678 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185682 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185686 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185693 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185699 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:27.187622 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185704 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185710 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185715 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185719 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185723 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185727 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185732 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185736 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185740 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185744 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185749 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185753 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185757 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185762 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185766 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185771 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185775 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185780 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185784 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185789 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:27.188468 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185794 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185799 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185803 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.185810 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187904 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187915 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187920 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187925 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187930 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187934 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187938 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187942 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187946 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187951 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187957 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187961 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187965 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187970 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187974 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187979 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:27.189270 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187984 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187988 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187992 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.187997 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188003 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188010 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188015 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188020 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188024 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188029 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188034 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188039 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188044 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188048 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188052 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188057 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188061 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188065 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188069 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188074 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:27.190149 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188078 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188083 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188087 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188091 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188095 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188099 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188104 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188109 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188113 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188118 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188122 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188126 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188130 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188134 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188138 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188142 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188146 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188151 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188155 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:27.190652 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188160 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188166 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188175 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188181 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188185 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188190 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188195 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188199 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188204 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188208 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188213 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188217 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188221 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188225 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188229 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188233 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188237 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188243 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188247 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:27.191434 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188252 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188256 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188262 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188266 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188270 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188275 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188279 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188284 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188288 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188292 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188297 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188301 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188305 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188309 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188313 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188319 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188323 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188328 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188333 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:27.192254 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188337 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188341 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188345 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188350 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188354 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188358 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188365 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188369 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188373 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188377 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188381 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188386 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.188406 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189302 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189313 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189320 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189326 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189331 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189336 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189340 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:27.193058 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189344 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189350 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189355 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189359 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189363 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189367 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189372 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189376 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189381 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189385 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189407 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189411 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189416 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189420 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189424 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189429 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189433 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189437 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189441 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189445 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:27.193641 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189450 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189453 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189458 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189462 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189466 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189470 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189474 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189479 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189485 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189489 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189495 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189501 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189505 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189510 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189514 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189519 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189523 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189527 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189531 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:27.194277 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189537 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189545 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189550 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189555 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189559 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189564 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189568 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189572 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189576 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189581 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189585 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189589 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189594 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189598 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189602 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189606 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189610 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189614 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189618 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189623 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:27.194763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189627 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189631 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189636 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189641 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189645 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189649 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189657 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189661 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189665 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189670 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189674 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189679 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189683 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189687 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189691 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189695 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189700 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189704 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189708 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189713 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:27.195317 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189717 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189722 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189726 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189730 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189734 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189739 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189743 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189748 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189752 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189757 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189761 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189765 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189769 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189774 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189778 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189783 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189787 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189790 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189796 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:27.195817 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.189800 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190722 2585 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190744 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190754 2585 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190762 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190769 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190776 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190789 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190795 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190800 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190806 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190811 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190817 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190822 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190827 2585 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190831 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190836 2585 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190841 2585 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190846 2585 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190853 2585 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190858 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190863 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190868 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190873 2585 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:14:27.196310 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190879 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190885 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190890 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190895 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190900 2585 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190906 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190911 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190918 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190922 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190930 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190935 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190941 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190946 2585 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190951 2585 flags.go:64] FLAG: --enable-server="true" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190956 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190962 2585 flags.go:64] FLAG: --event-burst="100" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190968 2585 flags.go:64] FLAG: --event-qps="50" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190973 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190977 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190982 2585 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190989 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190994 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.190999 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191004 2585 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191009 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:14:27.196895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191014 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191018 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191023 2585 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191028 2585 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191033 2585 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191038 2585 flags.go:64] FLAG: --feature-gates="" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191045 2585 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191049 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191055 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191060 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191065 2585 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191070 2585 flags.go:64] FLAG: --help="false" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191075 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191081 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191087 2585 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191092 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191098 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191104 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191110 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191115 2585 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191119 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191124 2585 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191129 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191134 2585 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:14:27.197550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191139 2585 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191144 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191148 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191153 2585 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191158 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191162 2585 flags.go:64] FLAG: --lock-file="" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191167 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191172 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191176 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191186 2585 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191190 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191195 2585 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191200 2585 flags.go:64] FLAG: --logging-format="text" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191205 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191210 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191215 2585 flags.go:64] FLAG: --manifest-url="" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191219 2585 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191226 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191231 2585 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191237 2585 flags.go:64] FLAG: --max-pods="110" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191242 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191247 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191254 2585 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191259 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191264 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:14:27.198215 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191269 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191275 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191287 2585 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191292 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191297 2585 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191302 2585 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191307 2585 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191312 2585 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191317 2585 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191321 2585 flags.go:64] FLAG: --port="10250" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191326 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191331 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0248937e7c6f384c7" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191336 2585 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191341 2585 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191346 2585 flags.go:64] FLAG: --register-node="true" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191351 2585 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191357 2585 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191362 2585 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191366 2585 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191371 2585 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191378 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191382 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191388 2585 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191407 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191412 2585 flags.go:64] FLAG: --runonce="false" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191417 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:14:27.198856 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191422 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191427 2585 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191431 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191438 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191444 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191449 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191453 2585 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191461 2585 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191466 2585 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191471 2585 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191476 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191481 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191486 2585 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191490 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191499 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191503 2585 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191508 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191516 2585 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191521 2585 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191526 2585 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191530 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191535 2585 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191540 2585 flags.go:64] FLAG: --v="2" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191547 2585 flags.go:64] FLAG: --version="false" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191554 2585 flags.go:64] FLAG: --vmodule="" Oct 29 22:14:27.199472 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191560 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.191566 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191777 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191783 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191788 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191792 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191796 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191800 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191805 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191809 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191815 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191820 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191827 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191832 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191839 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191843 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191847 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191852 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191856 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:27.200048 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191861 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191866 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191870 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191874 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191878 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191882 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191887 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191891 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191895 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191900 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191904 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191908 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191913 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191917 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191922 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191927 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191931 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191935 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191940 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191944 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:27.200579 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191949 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191953 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191957 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191963 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191967 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191971 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191976 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191982 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191986 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191990 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191994 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.191999 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192004 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192008 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192013 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192017 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192021 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192025 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192030 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192035 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:27.201081 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192039 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192043 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192048 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192052 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192056 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192061 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192065 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192069 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192073 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192078 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192082 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192087 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192092 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192096 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192103 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192112 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192117 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192122 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192126 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:27.201571 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192133 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192138 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192142 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192147 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192151 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192155 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192159 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192164 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192168 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192172 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192177 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192181 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192186 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192190 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192195 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192199 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192203 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192207 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192212 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192216 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:27.202009 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192221 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192225 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192230 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192234 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192238 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192242 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192246 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192250 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192257 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192260 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.192301 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192372 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192380 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192406 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192412 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192416 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:27.202500 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192420 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192425 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192430 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192434 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192438 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192443 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192447 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192451 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192455 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192467 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192471 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192475 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192480 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192484 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192488 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192492 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192497 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192501 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192505 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192510 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:27.202878 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192514 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192518 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192522 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192526 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192531 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192535 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192542 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192547 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192552 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192558 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192564 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192568 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192573 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192577 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192581 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192586 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192590 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192594 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192598 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:27.203339 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192602 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192606 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192610 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192615 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192619 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192623 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192627 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192631 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192636 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192640 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192645 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192649 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192653 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192657 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192662 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192666 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192670 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192675 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192679 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192684 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:27.203797 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192688 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192693 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192698 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192703 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192708 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192712 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192716 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192721 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192725 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192730 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192734 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192738 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192742 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192748 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192752 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192756 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192760 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192764 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192769 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192773 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:27.204267 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192778 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192782 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192786 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192791 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192795 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192799 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192804 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192808 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192815 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192822 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192828 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192833 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192837 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192841 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192847 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192852 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192856 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192861 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192866 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:27.204763 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192870 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192874 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.192878 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.199966 2585 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.199982 2585 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.199998 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.200002 2585 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200063 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200069 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200072 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200076 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200079 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200081 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200084 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200087 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200090 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200093 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200095 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200098 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:27.205208 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200101 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200103 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200106 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200109 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200112 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200116 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200120 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200123 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200126 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200128 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200131 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200133 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200136 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200138 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200141 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200143 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200147 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200151 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200153 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200156 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:27.205706 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200159 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200162 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200164 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200167 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200171 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200175 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200178 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200181 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200184 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200186 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200189 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200191 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200194 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200197 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200199 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200202 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200204 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200207 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200210 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:27.206174 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200212 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200215 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200217 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200220 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200222 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200225 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200228 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200231 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200234 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200236 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200239 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200242 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200245 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200248 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200251 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200253 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200256 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200258 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200261 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200264 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:27.206667 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200266 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200269 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200271 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200274 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200277 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200279 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200282 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200284 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200287 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200290 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200292 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200295 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200298 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200300 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200303 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200305 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200308 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200311 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200313 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:27.207119 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200316 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200319 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200321 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200324 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200327 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200330 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200333 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200335 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200338 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200340 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200343 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200345 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200347 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200350 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200353 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200356 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:27.207575 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.200383 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200545 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200550 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200552 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200555 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200558 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200561 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200563 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200566 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200569 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200571 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200574 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200576 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200579 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200582 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200585 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200588 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200591 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200593 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200596 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:27.207945 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200598 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200601 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200604 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200607 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200609 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200612 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200614 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200618 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200621 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200624 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200627 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200630 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200632 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200635 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200637 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200640 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200642 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200645 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200647 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:27.208435 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200650 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200652 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200655 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200658 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200660 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200663 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200666 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200668 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200671 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200673 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200676 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200678 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200681 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200683 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200686 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200689 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200692 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200695 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200697 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:27.209032 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200700 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200702 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200705 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200707 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200709 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200712 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200714 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200717 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200719 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200722 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200724 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200726 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200730 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200734 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200737 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200744 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200747 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200749 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200752 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:27.209682 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200754 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200757 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200760 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200762 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200765 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200767 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200770 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200772 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200775 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200777 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200780 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200783 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200786 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200788 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200790 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200793 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200796 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200798 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200801 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200803 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:27.210158 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200806 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200808 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200811 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200813 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200816 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200818 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200820 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200823 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200827 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200830 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.200853 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200883 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200887 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200890 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200894 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200896 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:14:27.210867 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200899 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200902 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200904 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200907 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200909 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200912 2585 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200914 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200916 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200920 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200922 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200925 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200928 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200930 2585 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200933 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200935 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200938 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200940 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200943 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200945 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200948 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:14:27.211314 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200950 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200953 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200955 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200958 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200961 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200964 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200966 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200969 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200971 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200974 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200977 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200979 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200983 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200987 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200990 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200993 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200995 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.200998 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201000 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:14:27.211838 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201003 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201005 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201024 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201028 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201032 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201035 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201037 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201040 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201043 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201045 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201048 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201051 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201053 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201056 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201058 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201061 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201063 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201067 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201070 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:14:27.212341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201072 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201076 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201078 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201081 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201084 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201087 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201090 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201092 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201095 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201098 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201100 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201103 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201105 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201108 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201110 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201113 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201115 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201120 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201123 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:14:27.213051 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201126 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201128 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201131 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201134 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201136 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201139 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201141 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201144 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201146 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201149 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201151 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201155 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201158 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201160 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201164 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201167 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201171 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201174 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201176 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:14:27.213515 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201179 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201181 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201184 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201186 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:27.201189 2585 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.201242 2585 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.203459 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.204256 2585 server.go:1014] "Starting client certificate rotation" Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.204348 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.204386 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.206022 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:14:27.214023 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.207071 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:14:27.217530 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.217513 2585 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:14:27.220593 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.220578 2585 log.go:25] "Validated CRI v1 image API" Oct 29 22:14:27.221316 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.221294 2585 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:14:27.223717 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.223694 2585 fs.go:135] Filesystem UUIDs: map[5a12698b-df65-4e0a-8d1f-b21dc75069d0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ec15bfe1-5bd9-4324-b832-4a4297747a3c:/dev/nvme0n1p4] Oct 29 22:14:27.223768 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.223717 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 29 22:14:27.229503 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.229382 2585 manager.go:217] Machine: {Timestamp:2026-10-29 22:14:27.227362091 +0000 UTC m=+0.302679390 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103936 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c9602487d30bccfadc9bf11dfbebf SystemUUID:ec2c9602-487d-30bc-cfad-c9bf11dfbebf BootID:c5a5a7f6-a179-45c7-a75d-f608ba40aace Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:a9:ac:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:a9:ac:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:94:9e:07:7c:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:14:27.229503 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.229502 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:14:27.229605 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.229576 2585 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:14:27.231783 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.231759 2585 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:14:27.231913 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.231784 2585 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:14:27.231972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.231914 2585 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:14:27.231972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.231923 2585 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:14:27.231972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.231956 2585 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:14:27.232051 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.231974 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:14:27.232781 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.232771 2585 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:14:27.236748 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.236736 2585 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:14:27.236879 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.236870 2585 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:14:27.237490 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.237480 2585 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:14:27.237520 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.237496 2585 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:14:27.237520 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.237508 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:14:27.237520 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.237518 2585 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:14:27.237622 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.237527 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:14:27.237862 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.237851 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:14:27.239811 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.239794 2585 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:14:27.244644 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.244631 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:14:27.245454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.245441 2585 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:14:27.245501 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.245472 2585 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:14:27.245617 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.245607 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:14:27.245654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.245623 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:14:27.245654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.245629 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:14:27.245654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.245635 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:14:27.245654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.245641 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:14:27.245654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.245648 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:14:27.246161 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.246152 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:14:27.246189 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.246162 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:14:27.246189 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.246169 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:14:27.246189 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.246175 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:14:27.246189 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.246183 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:14:27.246306 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.246191 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:14:27.246582 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.246572 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:14:27.246582 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.246582 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:14:27.252054 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.252037 2585 server.go:1263] "Started kubelet" Oct 29 22:14:27.252185 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.252138 2585 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:14:27.252230 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.252131 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:14:27.252230 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.252219 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:14:27.252777 ip-10-0-133-251 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:14:27.254038 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.254025 2585 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:14:27.254451 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.254436 2585 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:14:27.258658 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.258639 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:14:27.259139 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.259118 2585 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:14:27.259815 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.259795 2585 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:14:27.259876 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.259818 2585 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:14:27.259929 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.259919 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:14:27.259990 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.259981 2585 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:14:27.260020 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.259991 2585 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:14:27.260650 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.260630 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found" Oct 29 22:14:27.260851 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.260817 2585 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 29 22:14:27.262614 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.262599 2585 factory.go:55] Registering systemd factory Oct 29 22:14:27.262683 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.262632 2585 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:14:27.262861 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.262849 2585 factory.go:153] Registering CRI-O factory Oct 29 22:14:27.262895 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.262864 2585 factory.go:223] Registration of the crio container factory successfully Oct 29 22:14:27.262926 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.262908 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:14:27.262958 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.262929 2585 factory.go:103] Registering Raw factory Oct 29 22:14:27.262958 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.262951 2585 manager.go:1199] Started watching for new ooms in manager Oct 29 22:14:27.263794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.263778 2585 manager.go:319] Starting recovery of all containers Oct 29 22:14:27.273622 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.273600 2585 manager.go:324] Recovery completed Oct 29 22:14:27.279885 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.279867 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:27.281819 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.281804 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:27.281882 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.281832 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:27.281882 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.281847 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:27.282519 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.282425 2585 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:14:27.282519 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.282437 2585 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:14:27.282519 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.282455 2585 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:14:27.282519 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.282492 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:27.283192 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.283175 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:27.283264 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.283208 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:27.283264 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.283218 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:27.284636 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.284623 2585 policy_none.go:50] "Start" Oct 29 22:14:27.284701 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.284639 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:14:27.284701 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.284648 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:14:27.285270 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.285253 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 29 22:14:27.285343 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.285253 2585 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 29 22:14:27.286412 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.285414 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-251.ec2.internal.18e31fd2e3fa13f0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-251.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-251.ec2.internal,},FirstTimestamp:2026-10-29 22:14:27.251999728 +0000 UTC m=+0.327317027,LastTimestamp:2026-10-29 22:14:27.251999728 +0000 UTC m=+0.327317027,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-251.ec2.internal,}" Oct 29 22:14:27.286563 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.286551 2585 policy_none.go:44] "Start" Oct 29 22:14:27.299781 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.299586 2585 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 29 22:14:27.301506 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.301486 2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:14:27.301573 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.301536 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qt2g2" Oct 29 22:14:27.309973 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.309948 2585 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:14:27.312548 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.312535 2585 manager.go:341] "Starting Device Plugin manager" Oct 29 22:14:27.312648 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.312566 2585 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:14:27.312648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.312578 2585 server.go:87] "Starting device plugin registration server" Oct 29 22:14:27.312822 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.312809 2585 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:14:27.312875 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.312823 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:14:27.312967 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.312951 2585 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:14:27.313087 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.313028 2585 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:14:27.313087 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.313035 2585 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:14:27.314242 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.314224 2585 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:14:27.314321 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.314273 2585 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-251.ec2.internal\" not found" Oct 29 22:14:27.329082 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.329066 2585 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:27.355868 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.352832 2585 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:27.370658 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.370623 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:14:27.371801 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.371779 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:14:27.371876 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.371809 2585 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:14:27.371876 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.371831 2585 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:14:27.371876 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:27.371869 2585 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 29 22:14:27.385642 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.385622 2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:27.413475 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.413457 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:14:27.414290 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.414276 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:14:27.414356 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.414304 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:14:27.414356 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.414317 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:14:27.414356 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.414341 2585 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.431492 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.431438 2585 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.472854 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.472824 2585 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"] Oct 29 22:14:27.474847 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.474831 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.476339 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.476320 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.496850 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.496834 2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.499979 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.499966 2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.518900 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.518868 2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:27.518980 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.518935 2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:14:27.561348 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.561327 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.561459 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.561352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.561459 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.561370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df1fead76acc43615f3bffca1a53f41e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"df1fead76acc43615f3bffca1a53f41e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.661558 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.661535 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.661558 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.661560 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.661743 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.661575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df1fead76acc43615f3bffca1a53f41e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"df1fead76acc43615f3bffca1a53f41e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.661743 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.661601 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df1fead76acc43615f3bffca1a53f41e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"df1fead76acc43615f3bffca1a53f41e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.661743 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.661622 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.661743 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.661630 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.823553 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.823470 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:14:27.823553 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:27.823541 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:14:28.203940 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.203859 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 29 22:14:28.204525 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.204055 2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:14:28.204525 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.204083 2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 29 22:14:28.237659 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.237635 2585 apiserver.go:52] "Watching apiserver" Oct 29 22:14:28.259683 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.259662 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 29 22:14:28.279529 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.279501 2585 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:14:28.279813 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.279796 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-lmmsc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal","kube-system/global-pull-secret-syncer-5trph","kube-system/konnectivity-agent-6v7fk","openshift-image-registry/node-ca-vhtpp","openshift-ovn-kubernetes/ovnkube-node-zflnw","kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal","openshift-multus/multus-additional-cni-plugins-dkbgr","openshift-multus/network-metrics-daemon-nmlxs","openshift-network-operator/iptables-alerter-84hqt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m","openshift-cluster-node-tuning-operator/tuned-x8plf","openshift-multus/multus-5rqc2","openshift-network-diagnostics/network-check-target-qq6r4"] Oct 29 22:14:28.281979 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.281960 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.283850 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.283831 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.283934 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.283897 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:28.286251 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.286232 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:28.288200 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.288180 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.290579 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.290560 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.292930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.292910 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.295634 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.295613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:28.295730 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.295682 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:28.300001 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.299981 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:14:28.300128 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.300112 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:14:28.300185 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.300127 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfkqw\"" Oct 29 22:14:28.302795 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.302777 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.305661 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.305645 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.305878 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.305827 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:09:28 +0000 UTC" deadline="2028-07-08 12:29:13.04373223 +0000 UTC" Oct 29 22:14:28.305878 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.305876 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14822h14m44.737860617s" Oct 29 22:14:28.308549 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.308534 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.311380 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.311363 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.314319 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.314298 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:28.314414 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.314366 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:28.319932 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.319914 2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 29 22:14:28.320093 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.319976 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-spzlp" Oct 29 22:14:28.362734 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.362705 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:14:28.364875 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.364845 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zddq\" (UniqueName: \"kubernetes.io/projected/b3b72e10-5245-4d8e-9699-bae1c061e336-kube-api-access-7zddq\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.364875 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.364884 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-lib-modules\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.365064 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.364974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-netns\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.365064 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-etc-selinux\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.365064 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365028 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-socket-dir-parent\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.365220 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365082 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-hostroot\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.365220 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365147 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-host\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.365220 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365189 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-konnectivity-ca\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:28.365371 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365222 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365668 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-tmp-dir\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365726 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-env-overrides\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365759 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-sys-fs\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365784 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-cnibin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365810 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-multus\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365838 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365864 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365939 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-var-lib-kubelet\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.365965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8md\" (UniqueName: \"kubernetes.io/projected/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-kube-api-access-sd8md\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.366062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.366011 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-ovn\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.366945 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.366035 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-socket-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.367026 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.366972 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-netd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.367026 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367000 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrr9\" (UniqueName: \"kubernetes.io/projected/629cfb60-8afd-46e7-86a8-1de336684f20-kube-api-access-9zrr9\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.367121 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367026 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrx4w\" (UniqueName: \"kubernetes.io/projected/e1daa8f6-7854-4c2a-b207-bd6947f3e030-kube-api-access-hrx4w\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.367121 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367050 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-etc-kubernetes\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.367121 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367108 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjvs\" (UniqueName: \"kubernetes.io/projected/cc7d661f-6060-41b9-a442-ab840f6129e5-kube-api-access-2wjvs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.367245 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367133 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj8m5\" (UniqueName: \"kubernetes.io/projected/4bfd4dd9-a2a6-4ecd-9667-840738787716-kube-api-access-pj8m5\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.367245 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367152 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-serviceca\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.367245 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367168 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-slash\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.367245 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367213 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.367421 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367243 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-system-cni-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.367421 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367270 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-systemd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.367421 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367303 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-bin\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.367421 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367354 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-config\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.367421 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367389 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-cni-binary-copy\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.367640 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367425 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-k8s-cni-cncf-io\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.367640 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367492 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-conf-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.367640 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-dbus\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.367640 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367567 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-conf\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.367640 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-run\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.367640 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367627 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-hosts-file\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.367834 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367655 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-host\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.367834 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367703 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-kubelet-config\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.367834 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367749 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-modprobe-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.367834 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367782 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-kubernetes\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.368021 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367818 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-node-log\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368021 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367868 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-device-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.368021 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367890 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-tmp\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.368021 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367932 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-log-socket\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368021 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.367974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368021 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368000 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-system-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.368021 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368022 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-cnibin\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.368313 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368039 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8769\" (UniqueName: \"kubernetes.io/projected/35dc91f0-4639-4294-a296-418c22a692c1-kube-api-access-d8769\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.368313 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368096 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-etc-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368313 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368143 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-multus-certs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.368313 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368195 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.368313 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368231 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-systemd\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.368313 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368260 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-registration-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.368313 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368277 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.368313 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368305 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-daemon-config\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.368648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368344 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysconfig\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.368648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368376 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-kubelet\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368424 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-netns\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.368648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368463 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.368648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368496 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:28.368648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368529 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n425l\" (UniqueName: \"kubernetes.io/projected/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-kube-api-access-n425l\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.368648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368614 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368648 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368639 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1daa8f6-7854-4c2a-b207-bd6947f3e030-host-slash\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368664 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/629cfb60-8afd-46e7-86a8-1de336684f20-ovn-node-metrics-cert\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368687 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-agent-certs\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368710 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shpll\" (UniqueName: \"kubernetes.io/projected/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-kube-api-access-shpll\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-tuned\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368757 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368780 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-kubelet\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368809 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368833 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-systemd-units\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368856 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-script-lib\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368879 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-bin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368901 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-sys\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.368930 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368925 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-var-lib-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.369383 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.368950 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1daa8f6-7854-4c2a-b207-bd6947f3e030-iptables-alerter-script\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.412986 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.412950 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:14:28.413104 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413009 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:14:28.413161 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413107 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:14:28.413214 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413171 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:28.413214 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413180 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:14:28.413279 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413220 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:14:28.413331 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413310 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:28.413331 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413320 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:14:28.413413 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413359 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:14:28.413671 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413651 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:14:28.413870 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413853 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2cw6s\"" Oct 29 22:14:28.413920 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413885 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kk6hd\"" Oct 29 22:14:28.413993 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413977 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:14:28.414049 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413985 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:14:28.414049 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.413978 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hznbv\"" Oct 29 22:14:28.414049 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414015 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:14:28.414049 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414026 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:14:28.414049 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414029 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:14:28.414246 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414090 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:14:28.414246 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414097 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:14:28.414246 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414101 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:14:28.414246 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414126 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:14:28.414246 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414135 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:14:28.414246 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.414158 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm48d\"" Oct 29 22:14:28.419257 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.419241 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:14:28.419447 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.419422 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:14:28.419512 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.419458 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pxx6w\"" Oct 29 22:14:28.427162 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.427146 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-475gp\"" Oct 29 22:14:28.427745 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.427727 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gbljw\"" Oct 29 22:14:28.427745 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.427737 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:14:28.427866 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.427727 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:14:28.427866 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.427732 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w24c4\"" Oct 29 22:14:28.427866 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.427789 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:14:28.462050 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.461939 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150171fc6f62800f31d26bdb941c8f1d.slice/crio-7eee30eb857c093ae68fcc631f38870418c1ea438d4d56ea3e50225dac2c09dd WatchSource:0}: Error finding container 7eee30eb857c093ae68fcc631f38870418c1ea438d4d56ea3e50225dac2c09dd: Status 404 returned error can't find the container with id 7eee30eb857c093ae68fcc631f38870418c1ea438d4d56ea3e50225dac2c09dd Oct 29 22:14:28.468848 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.468831 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:14:28.469563 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469535 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-env-overrides\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.469668 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-sys-fs\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.469668 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469600 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-cnibin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.469668 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469643 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-multus\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469670 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469713 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-sys-fs\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469713 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-cnibin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469749 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-var-lib-kubelet\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8md\" (UniqueName: \"kubernetes.io/projected/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-kube-api-access-sd8md\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.469810 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-ovn\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469838 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-socket-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469853 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-netd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.469880 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret podName:ab618391-4607-4f00-ab02-819dc9897da1 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:28.969855078 +0000 UTC m=+2.045172384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret") pod "global-pull-secret-syncer-5trph" (UID: "ab618391-4607-4f00-ab02-819dc9897da1") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469885 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-netd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469915 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrr9\" (UniqueName: \"kubernetes.io/projected/629cfb60-8afd-46e7-86a8-1de336684f20-kube-api-access-9zrr9\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.469942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrx4w\" (UniqueName: \"kubernetes.io/projected/e1daa8f6-7854-4c2a-b207-bd6947f3e030-kube-api-access-hrx4w\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.469966 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.469975 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-etc-kubernetes\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.470006 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs podName:b4a2ad5d-ac31-4ccf-8cd0-302df20209c0 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:28.969993722 +0000 UTC m=+2.045311044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs") pod "network-metrics-daemon-nmlxs" (UID: "b4a2ad5d-ac31-4ccf-8cd0-302df20209c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470020 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjvs\" (UniqueName: \"kubernetes.io/projected/cc7d661f-6060-41b9-a442-ab840f6129e5-kube-api-access-2wjvs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470032 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-etc-kubernetes\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470061 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8m5\" (UniqueName: \"kubernetes.io/projected/4bfd4dd9-a2a6-4ecd-9667-840738787716-kube-api-access-pj8m5\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470082 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-multus\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470078 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-serviceca\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470115 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-slash\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470139 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470156 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-system-cni-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-systemd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470194 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-bin\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470210 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-config\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-cni-binary-copy\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470241 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-k8s-cni-cncf-io\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.470794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470256 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-conf-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470272 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-dbus\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-conf\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470333 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-run\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470359 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-hosts-file\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-host\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470415 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-kubelet-config\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470441 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-modprobe-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470453 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-kubernetes\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470538 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-node-log\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470547 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-kubernetes\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-device-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470603 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-bin\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-tmp\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470624 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-device-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470642 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-log-socket\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.471679 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470675 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-node-log\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470679 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470715 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-system-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470744 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-cnibin\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470769 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-env-overrides\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470774 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8769\" (UniqueName: \"kubernetes.io/projected/35dc91f0-4639-4294-a296-418c22a692c1-kube-api-access-d8769\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-etc-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470837 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-multus-certs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470866 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470893 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-systemd\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-registration-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471011 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-daemon-config\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471039 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysconfig\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471065 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-kubelet\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471091 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-netns\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471150 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-slash\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.472545 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471215 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-hosts-file\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471218 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-kubelet-config\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471239 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-serviceca\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471283 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471298 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-ovn\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471308 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-modprobe-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471155 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-netns\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471340 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-system-cni-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471352 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-host\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471388 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-systemd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471400 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-var-lib-kubelet\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471429 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-config\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471448 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-multus-certs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471497 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-socket-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471514 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-log-socket\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471578 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-system-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471609 2585 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471624 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-cnibin\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.473454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471816 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471849 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471875 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n425l\" (UniqueName: \"kubernetes.io/projected/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-kube-api-access-n425l\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471931 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1daa8f6-7854-4c2a-b207-bd6947f3e030-host-slash\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471936 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-etc-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.470811 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-run\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471956 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/629cfb60-8afd-46e7-86a8-1de336684f20-ovn-node-metrics-cert\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-agent-certs\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.471997 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shpll\" (UniqueName: \"kubernetes.io/projected/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-kube-api-access-shpll\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472013 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-tuned\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472010 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-cni-binary-copy\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472029 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472063 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472066 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-kubelet\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472133 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-systemd-units\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.474283 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472156 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-script-lib\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472184 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-bin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472213 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-sys\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472243 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-var-lib-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1daa8f6-7854-4c2a-b207-bd6947f3e030-iptables-alerter-script\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472297 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zddq\" (UniqueName: \"kubernetes.io/projected/b3b72e10-5245-4d8e-9699-bae1c061e336-kube-api-access-7zddq\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472301 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472323 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-lib-modules\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-netns\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472352 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1daa8f6-7854-4c2a-b207-bd6947f3e030-host-slash\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-etc-selinux\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472462 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-socket-dir-parent\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472505 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-hostroot\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472515 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-host\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472571 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-konnectivity-ca\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472590 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-host\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472602 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472630 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-tmp-dir\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472638 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-k8s-cni-cncf-io\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.472705 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-systemd-units\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.473156 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.473306 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-script-lib\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.473365 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-bin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.473416 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-sys\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.473492 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-var-lib-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.473783 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-konnectivity-ca\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.473893 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.473971 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1daa8f6-7854-4c2a-b207-bd6947f3e030-iptables-alerter-script\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474196 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474577 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysconfig\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474613 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-conf\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474620 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474725 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-tmp-dir\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474760 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-lib-modules\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.475909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474784 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-kubelet\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474797 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-kubelet\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474804 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-netns\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.474844 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-conf-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475006 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-dbus\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475071 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-systemd\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475102 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-etc-selinux\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475120 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-registration-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475125 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475132 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-daemon-config\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475159 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-socket-dir-parent\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475163 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-hostroot\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475368 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/629cfb60-8afd-46e7-86a8-1de336684f20-ovn-node-metrics-cert\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.476716 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.475629 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-tmp\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.477159 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.477000 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-tuned\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.477774 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.477758 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-agent-certs\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:28.492705 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.492676 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:28.492818 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.492710 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:28.492818 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.492723 2585 projected.go:196] Error preparing data for projected volume kube-api-access-zn6ww for pod openshift-network-diagnostics/network-check-target-qq6r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:28.492818 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.492778 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww podName:1d3f4dde-95c6-41df-951b-316cd9d10f0a nodeName:}" failed. No retries permitted until 2026-10-29 22:14:28.992760759 +0000 UTC m=+2.068078065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zn6ww" (UniqueName: "kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww") pod "network-check-target-qq6r4" (UID: "1d3f4dde-95c6-41df-951b-316cd9d10f0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:28.493383 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.493273 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8md\" (UniqueName: \"kubernetes.io/projected/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-kube-api-access-sd8md\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.493383 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.493283 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8m5\" (UniqueName: \"kubernetes.io/projected/4bfd4dd9-a2a6-4ecd-9667-840738787716-kube-api-access-pj8m5\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.493383 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.493337 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n425l\" (UniqueName: \"kubernetes.io/projected/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-kube-api-access-n425l\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.493383 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.493348 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrx4w\" (UniqueName: \"kubernetes.io/projected/e1daa8f6-7854-4c2a-b207-bd6947f3e030-kube-api-access-hrx4w\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.493621 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.493500 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrr9\" (UniqueName: \"kubernetes.io/projected/629cfb60-8afd-46e7-86a8-1de336684f20-kube-api-access-9zrr9\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.493621 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.493510 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjvs\" (UniqueName: \"kubernetes.io/projected/cc7d661f-6060-41b9-a442-ab840f6129e5-kube-api-access-2wjvs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.493707 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.493690 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8769\" (UniqueName: \"kubernetes.io/projected/35dc91f0-4639-4294-a296-418c22a692c1-kube-api-access-d8769\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.494909 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.494892 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zddq\" (UniqueName: \"kubernetes.io/projected/b3b72e10-5245-4d8e-9699-bae1c061e336-kube-api-access-7zddq\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.495273 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.495257 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shpll\" (UniqueName: \"kubernetes.io/projected/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-kube-api-access-shpll\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:28.618006 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.617971 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:14:28.622550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.622521 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:28.629126 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.629107 2585 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:28.629714 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.629694 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bb246b_2e00_4a19_a53f_ffcc9d8c0c79.slice/crio-d428635318497c8cd054862e7ff80171b2e5accdc73366c679a8bfa7f91dd5de WatchSource:0}: Error finding container d428635318497c8cd054862e7ff80171b2e5accdc73366c679a8bfa7f91dd5de: Status 404 returned error can't find the container with id d428635318497c8cd054862e7ff80171b2e5accdc73366c679a8bfa7f91dd5de Oct 29 22:14:28.632618 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.632597 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa39a765_7c7f_4212_a8e2_c7f2a91f7339.slice/crio-abaf932855fdfcd88bf7d4c28ff3df7833181ea8cbe046a483d1c7a6e48869be WatchSource:0}: Error finding container abaf932855fdfcd88bf7d4c28ff3df7833181ea8cbe046a483d1c7a6e48869be: Status 404 returned error can't find the container with id abaf932855fdfcd88bf7d4c28ff3df7833181ea8cbe046a483d1c7a6e48869be Oct 29 22:14:28.651574 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.651547 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:14:28.661341 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.661305 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4b66ac_6aa8_4ed0_b1d4_0c9b7fc36d09.slice/crio-558717eb74111b51c72a34af66623138e8f8d2d6cf6bbee0dddfe1bd309de5d4 WatchSource:0}: Error finding container 558717eb74111b51c72a34af66623138e8f8d2d6cf6bbee0dddfe1bd309de5d4: Status 404 returned error can't find the container with id 558717eb74111b51c72a34af66623138e8f8d2d6cf6bbee0dddfe1bd309de5d4 Oct 29 22:14:28.681085 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.681066 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:28.689780 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.689756 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629cfb60_8afd_46e7_86a8_1de336684f20.slice/crio-22b3d2a8e71ef4dfe0ffae07c71ca79169936720dd8fd257539805f3c409cea6 WatchSource:0}: Error finding container 22b3d2a8e71ef4dfe0ffae07c71ca79169936720dd8fd257539805f3c409cea6: Status 404 returned error can't find the container with id 22b3d2a8e71ef4dfe0ffae07c71ca79169936720dd8fd257539805f3c409cea6 Oct 29 22:14:28.699177 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.699159 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:14:28.700743 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.700712 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:14:28.708649 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.708627 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1daa8f6_7854_4c2a_b207_bd6947f3e030.slice/crio-705c2f09cd0cbcdf4d8830dfaa7b519fdea6cebe904bc25737de615723ad0ed8 WatchSource:0}: Error finding container 705c2f09cd0cbcdf4d8830dfaa7b519fdea6cebe904bc25737de615723ad0ed8: Status 404 returned error can't find the container with id 705c2f09cd0cbcdf4d8830dfaa7b519fdea6cebe904bc25737de615723ad0ed8 Oct 29 22:14:28.710792 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.710770 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:14:28.711679 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.711658 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dc91f0_4639_4294_a296_418c22a692c1.slice/crio-fa41961e658bac9ac8d71a1ff36164e523e435257775628b5b2bee14e1ed55fc WatchSource:0}: Error finding container fa41961e658bac9ac8d71a1ff36164e523e435257775628b5b2bee14e1ed55fc: Status 404 returned error can't find the container with id fa41961e658bac9ac8d71a1ff36164e523e435257775628b5b2bee14e1ed55fc Oct 29 22:14:28.713327 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.713313 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:14:28.720897 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.720873 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b72e10_5245_4d8e_9699_bae1c061e336.slice/crio-105e933c13d5663a26543f97ec22b92ca3bb2edef90ab4abea7c5d64fe1eadf2 WatchSource:0}: Error finding container 105e933c13d5663a26543f97ec22b92ca3bb2edef90ab4abea7c5d64fe1eadf2: Status 404 returned error can't find the container with id 105e933c13d5663a26543f97ec22b92ca3bb2edef90ab4abea7c5d64fe1eadf2 Oct 29 22:14:28.723557 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.723537 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfd4dd9_a2a6_4ecd_9667_840738787716.slice/crio-9904ff5f8a1236580d5d8ba38edd7034906966176acd942870b9cbcd1b3751f0 WatchSource:0}: Error finding container 9904ff5f8a1236580d5d8ba38edd7034906966176acd942870b9cbcd1b3751f0: Status 404 returned error can't find the container with id 9904ff5f8a1236580d5d8ba38edd7034906966176acd942870b9cbcd1b3751f0 Oct 29 22:14:28.728441 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.728423 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5rqc2" Oct 29 22:14:28.738378 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:14:28.738357 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7d661f_6060_41b9_a442_ab840f6129e5.slice/crio-fca0ff5a75c41a1042824b1c9cacd3386cfada13220a4c74a25ad1f0a19d1764 WatchSource:0}: Error finding container fca0ff5a75c41a1042824b1c9cacd3386cfada13220a4c74a25ad1f0a19d1764: Status 404 returned error can't find the container with id fca0ff5a75c41a1042824b1c9cacd3386cfada13220a4c74a25ad1f0a19d1764 Oct 29 22:14:28.975889 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.975657 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:28.975889 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:28.975705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:28.975889 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.975810 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:28.975889 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.975811 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:28.975889 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.975865 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret podName:ab618391-4607-4f00-ab02-819dc9897da1 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:29.975848386 +0000 UTC m=+3.051165688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret") pod "global-pull-secret-syncer-5trph" (UID: "ab618391-4607-4f00-ab02-819dc9897da1") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:28.976220 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:28.975924 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs podName:b4a2ad5d-ac31-4ccf-8cd0-302df20209c0 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:29.975900072 +0000 UTC m=+3.051217359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs") pod "network-metrics-daemon-nmlxs" (UID: "b4a2ad5d-ac31-4ccf-8cd0-302df20209c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:29.076490 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.076448 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:29.076688 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.076635 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:29.076688 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.076650 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:29.076688 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.076659 2585 projected.go:196] Error preparing data for projected volume kube-api-access-zn6ww for pod openshift-network-diagnostics/network-check-target-qq6r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:29.076848 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.076705 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww podName:1d3f4dde-95c6-41df-951b-316cd9d10f0a nodeName:}" failed. No retries permitted until 2026-10-29 22:14:30.076692462 +0000 UTC m=+3.152009764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zn6ww" (UniqueName: "kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww") pod "network-check-target-qq6r4" (UID: "1d3f4dde-95c6-41df-951b-316cd9d10f0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:29.323745 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.323560 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:29 +0000 UTC" deadline="2028-05-23 15:08:24.665518107 +0000 UTC" Oct 29 22:14:29.323745 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.323636 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13720h53m55.341887412s" Oct 29 22:14:29.375087 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.372613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:29.375087 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.372749 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:29.413415 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.413342 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" event={"ID":"35dc91f0-4639-4294-a296-418c22a692c1","Type":"ContainerStarted","Data":"fa41961e658bac9ac8d71a1ff36164e523e435257775628b5b2bee14e1ed55fc"} Oct 29 22:14:29.428597 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.428523 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-84hqt" event={"ID":"e1daa8f6-7854-4c2a-b207-bd6947f3e030","Type":"ContainerStarted","Data":"705c2f09cd0cbcdf4d8830dfaa7b519fdea6cebe904bc25737de615723ad0ed8"} Oct 29 22:14:29.431102 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.431006 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vhtpp" event={"ID":"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09","Type":"ContainerStarted","Data":"558717eb74111b51c72a34af66623138e8f8d2d6cf6bbee0dddfe1bd309de5d4"} Oct 29 22:14:29.444817 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.444576 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6v7fk" event={"ID":"aa39a765-7c7f-4212-a8e2-c7f2a91f7339","Type":"ContainerStarted","Data":"abaf932855fdfcd88bf7d4c28ff3df7833181ea8cbe046a483d1c7a6e48869be"} Oct 29 22:14:29.462259 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.462218 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" event={"ID":"df1fead76acc43615f3bffca1a53f41e","Type":"ContainerStarted","Data":"7931779220d01a3c9e6706d456b9431572ac5cb47009367ae9fb0f5c43541e8a"} Oct 29 22:14:29.467516 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.466568 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"22b3d2a8e71ef4dfe0ffae07c71ca79169936720dd8fd257539805f3c409cea6"} Oct 29 22:14:29.468587 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.468565 2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:29.477328 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.477297 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lmmsc" event={"ID":"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79","Type":"ContainerStarted","Data":"d428635318497c8cd054862e7ff80171b2e5accdc73366c679a8bfa7f91dd5de"} Oct 29 22:14:29.486709 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.486677 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"150171fc6f62800f31d26bdb941c8f1d","Type":"ContainerStarted","Data":"7eee30eb857c093ae68fcc631f38870418c1ea438d4d56ea3e50225dac2c09dd"} Oct 29 22:14:29.508643 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.508604 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5rqc2" event={"ID":"cc7d661f-6060-41b9-a442-ab840f6129e5","Type":"ContainerStarted","Data":"fca0ff5a75c41a1042824b1c9cacd3386cfada13220a4c74a25ad1f0a19d1764"} Oct 29 22:14:29.518457 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.518422 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x8plf" event={"ID":"4bfd4dd9-a2a6-4ecd-9667-840738787716","Type":"ContainerStarted","Data":"9904ff5f8a1236580d5d8ba38edd7034906966176acd942870b9cbcd1b3751f0"} Oct 29 22:14:29.527196 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.527160 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" event={"ID":"b3b72e10-5245-4d8e-9699-bae1c061e336","Type":"ContainerStarted","Data":"105e933c13d5663a26543f97ec22b92ca3bb2edef90ab4abea7c5d64fe1eadf2"} Oct 29 22:14:29.720341 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.720251 2585 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:14:29.992536 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.992433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:29.992536 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:29.992496 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:29.992849 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.992712 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:29.992849 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.992780 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs podName:b4a2ad5d-ac31-4ccf-8cd0-302df20209c0 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:31.992761483 +0000 UTC m=+5.068078773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs") pod "network-metrics-daemon-nmlxs" (UID: "b4a2ad5d-ac31-4ccf-8cd0-302df20209c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:29.993508 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.993214 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:29.993508 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:29.993275 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret podName:ab618391-4607-4f00-ab02-819dc9897da1 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:31.993258343 +0000 UTC m=+5.068575633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret") pod "global-pull-secret-syncer-5trph" (UID: "ab618391-4607-4f00-ab02-819dc9897da1") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:30.094450 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:30.093733 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:30.094450 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:30.093962 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:30.094450 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:30.093986 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:30.094450 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:30.094000 2585 projected.go:196] Error preparing data for projected volume kube-api-access-zn6ww for pod openshift-network-diagnostics/network-check-target-qq6r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:30.094450 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:30.094062 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww podName:1d3f4dde-95c6-41df-951b-316cd9d10f0a nodeName:}" failed. No retries permitted until 2026-10-29 22:14:32.094042575 +0000 UTC m=+5.169359872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zn6ww" (UniqueName: "kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww") pod "network-check-target-qq6r4" (UID: "1d3f4dde-95c6-41df-951b-316cd9d10f0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:30.324842 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:30.324690 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:29 +0000 UTC" deadline="2028-05-03 17:55:42.379871453 +0000 UTC" Oct 29 22:14:30.324842 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:30.324754 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13243h41m12.055121793s" Oct 29 22:14:30.373549 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:30.372979 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:30.373549 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:30.373132 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:30.373800 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:30.373607 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:30.373800 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:30.373736 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:31.375265 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:31.374733 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:31.375265 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:31.374874 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:32.012748 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:32.012718 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:32.012857 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:32.012767 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:32.012979 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.012939 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:32.013034 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.013000 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs podName:b4a2ad5d-ac31-4ccf-8cd0-302df20209c0 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.012982043 +0000 UTC m=+9.088299333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs") pod "network-metrics-daemon-nmlxs" (UID: "b4a2ad5d-ac31-4ccf-8cd0-302df20209c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:32.013099 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.013069 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:32.013202 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.013109 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret podName:ab618391-4607-4f00-ab02-819dc9897da1 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.013095918 +0000 UTC m=+9.088413222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret") pod "global-pull-secret-syncer-5trph" (UID: "ab618391-4607-4f00-ab02-819dc9897da1") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:32.113583 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:32.113548 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:32.113786 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.113760 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:32.113786 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.113784 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:32.113893 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.113799 2585 projected.go:196] Error preparing data for projected volume kube-api-access-zn6ww for pod openshift-network-diagnostics/network-check-target-qq6r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:32.113893 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.113861 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww podName:1d3f4dde-95c6-41df-951b-316cd9d10f0a nodeName:}" failed. No retries permitted until 2026-10-29 22:14:36.113841877 +0000 UTC m=+9.189159180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zn6ww" (UniqueName: "kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww") pod "network-check-target-qq6r4" (UID: "1d3f4dde-95c6-41df-951b-316cd9d10f0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:32.372513 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:32.372478 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:32.372685 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.372647 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:32.372759 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:32.372713 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:32.372818 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:32.372785 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:33.372748 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.372136 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:33.372748 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:33.372430 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:33.594905 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.593615 2585 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="afc510cc36cdd11a57e8d8d7e2dd40908806042ddad3463e285bccfba45518f0" exitCode=0 Oct 29 22:14:33.594905 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.593726 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" event={"ID":"35dc91f0-4639-4294-a296-418c22a692c1","Type":"ContainerDied","Data":"afc510cc36cdd11a57e8d8d7e2dd40908806042ddad3463e285bccfba45518f0"} Oct 29 22:14:33.598569 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.597920 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6v7fk" event={"ID":"aa39a765-7c7f-4212-a8e2-c7f2a91f7339","Type":"ContainerStarted","Data":"60349e48d8f8c36157d5edde1935e4dbdd88333f285dbb575c359303c99f9aa1"} Oct 29 22:14:33.601028 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.600376 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" event={"ID":"df1fead76acc43615f3bffca1a53f41e","Type":"ContainerStarted","Data":"70e4f9a655b73db079b40320ea7478de2be4a0e224b7a23abb2f91042360e091"} Oct 29 22:14:33.605316 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.604689 2585 generic.go:356] "Generic (PLEG): container finished" podID="150171fc6f62800f31d26bdb941c8f1d" containerID="f2a86176bd78be0bacc7b2a4f21ce053b71ccfc2d68d443c11ef4f6d5569f9de" exitCode=0 Oct 29 22:14:33.605316 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.604732 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"150171fc6f62800f31d26bdb941c8f1d","Type":"ContainerDied","Data":"f2a86176bd78be0bacc7b2a4f21ce053b71ccfc2d68d443c11ef4f6d5569f9de"} Oct 29 22:14:33.738957 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.738201 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6v7fk" podStartSLOduration=1.10425029 podStartE2EDuration="5.738177548s" totalImagesPullingTime="4.633927258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:33.695709857 +0000 UTC m=+6.771027166" watchObservedRunningTime="2026-10-29 22:14:33.738177548 +0000 UTC m=+6.813494866" Oct 29 22:14:33.781640 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:33.781578 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" podStartSLOduration=5.781557912 podStartE2EDuration="5.781557912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:33.780612371 +0000 UTC m=+6.855929682" watchObservedRunningTime="2026-10-29 22:14:33.781557912 +0000 UTC m=+6.856875231" Oct 29 22:14:34.373350 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:34.372616 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:34.373350 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:34.372777 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:34.373350 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:34.373201 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:34.373350 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:34.373309 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:34.615062 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:34.615023 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"150171fc6f62800f31d26bdb941c8f1d","Type":"ContainerStarted","Data":"616acac671d6c5f7f9b09e479b731bab783800a4cd8780ad90aeb11876e8d55a"} Oct 29 22:14:34.627612 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:34.627536 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vhtpp" event={"ID":"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09","Type":"ContainerStarted","Data":"6e3176fba116b6b0504e2f718303846f5e6a66037318a3ac57cbeb769b263d50"} Oct 29 22:14:34.699049 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:34.698995 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vhtpp" podStartSLOduration=0.972614341 podStartE2EDuration="6.698978481s" totalImagesPullingTime="5.72636414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:34.698706355 +0000 UTC m=+7.774023666" watchObservedRunningTime="2026-10-29 22:14:34.698978481 +0000 UTC m=+7.774295792" Oct 29 22:14:34.699708 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:34.699662 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" podStartSLOduration=6.699649222 podStartE2EDuration="6.699649222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:34.65759752 +0000 UTC m=+7.732914834" watchObservedRunningTime="2026-10-29 22:14:34.699649222 +0000 UTC m=+7.774966535" Oct 29 22:14:35.375166 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:35.374669 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:35.375166 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:35.374813 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:36.056768 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:36.055986 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:36.056768 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:36.056044 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:36.056768 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.056180 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:36.056768 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.056246 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs podName:b4a2ad5d-ac31-4ccf-8cd0-302df20209c0 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:44.056227621 +0000 UTC m=+17.131544915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs") pod "network-metrics-daemon-nmlxs" (UID: "b4a2ad5d-ac31-4ccf-8cd0-302df20209c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:36.056768 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.056680 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:36.056768 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.056729 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret podName:ab618391-4607-4f00-ab02-819dc9897da1 nodeName:}" failed. No retries permitted until 2026-10-29 22:14:44.056714786 +0000 UTC m=+17.132032092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret") pod "global-pull-secret-syncer-5trph" (UID: "ab618391-4607-4f00-ab02-819dc9897da1") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:36.157592 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:36.156844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:36.157592 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.157072 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:36.157592 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.157095 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:36.157592 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.157110 2585 projected.go:196] Error preparing data for projected volume kube-api-access-zn6ww for pod openshift-network-diagnostics/network-check-target-qq6r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:36.157592 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.157174 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww podName:1d3f4dde-95c6-41df-951b-316cd9d10f0a nodeName:}" failed. No retries permitted until 2026-10-29 22:14:44.157154591 +0000 UTC m=+17.232471894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zn6ww" (UniqueName: "kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww") pod "network-check-target-qq6r4" (UID: "1d3f4dde-95c6-41df-951b-316cd9d10f0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:36.373458 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:36.372727 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:36.373458 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.372859 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:36.373458 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:36.373274 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:36.373458 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:36.373364 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:36.638643 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:36.638538 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" event={"ID":"b3b72e10-5245-4d8e-9699-bae1c061e336","Type":"ContainerStarted","Data":"54d9fe94e122a5b9e6653e33b457927b616dd97db7ee0c150677345580e4774a"} Oct 29 22:14:36.901896 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:36.901326 2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:36.902069 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:36.901934 2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:37.237504 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.237474 2585 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:14:37.321176 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.320424 2585 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:14:37.237498503Z","UUID":"d56ea1c0-9bf7-49c1-a8e0-8cbb443141ce","Handler":null,"Name":"","Endpoint":""} Oct 29 22:14:37.322528 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.322296 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:14:37.322528 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.322319 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:14:37.378563 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.378531 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:37.378757 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:37.378653 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:37.645496 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.645397 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5rqc2" event={"ID":"cc7d661f-6060-41b9-a442-ab840f6129e5","Type":"ContainerStarted","Data":"92287f1e6916a2881863598768d46d4ed835547f3f69433ad18202f8678306ef"} Oct 29 22:14:37.653435 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.650264 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" event={"ID":"b3b72e10-5245-4d8e-9699-bae1c061e336","Type":"ContainerStarted","Data":"15d2edd96a01f7829a3ebcc57c83ccc66ae35a6ff10ac6ff9493b4a74eee209f"} Oct 29 22:14:37.677221 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.677180 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:37.689395 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.689337 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5rqc2" podStartSLOduration=1.224969607 podStartE2EDuration="9.689315561s" totalImagesPullingTime="8.464345954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:37.689311831 +0000 UTC m=+10.764629148" watchObservedRunningTime="2026-10-29 22:14:37.689315561 +0000 UTC m=+10.764632885" Oct 29 22:14:37.839802 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:37.839766 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:38.372717 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:38.372687 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:38.372850 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:38.372827 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:38.372910 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:38.372866 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:38.372974 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:38.372956 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:38.655741 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:38.655660 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" event={"ID":"b3b72e10-5245-4d8e-9699-bae1c061e336","Type":"ContainerStarted","Data":"7f3877a103e611faa56cc1406ad4373d2eec30ce1445f69680e59b39ce3d8773"} Oct 29 22:14:38.657594 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:38.657561 2585 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="fc800720f8eab3c5536499d83fe671e02ef9fda465235bd0b4ea21c7053b16bf" exitCode=0 Oct 29 22:14:38.657737 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:38.657596 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" event={"ID":"35dc91f0-4639-4294-a296-418c22a692c1","Type":"ContainerDied","Data":"fc800720f8eab3c5536499d83fe671e02ef9fda465235bd0b4ea21c7053b16bf"} Oct 29 22:14:38.661508 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:38.660935 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lmmsc" event={"ID":"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79","Type":"ContainerStarted","Data":"a7a7b0fc8386a82dd32316ce029f91bf55eb071d4fc0334ae3fe2b4dbf927504"} Oct 29 22:14:38.700241 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:38.700184 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" podStartSLOduration=1.5136347620000001 podStartE2EDuration="10.700168081s" totalImagesPullingTime="9.186533319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:38.699249592 +0000 UTC m=+11.774566905" watchObservedRunningTime="2026-10-29 22:14:38.700168081 +0000 UTC m=+11.775485394" Oct 29 22:14:38.739990 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:38.739922 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lmmsc" podStartSLOduration=1.7785541729999998 podStartE2EDuration="10.739900993s" totalImagesPullingTime="8.96134682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:38.739840563 +0000 UTC m=+11.815157877" watchObservedRunningTime="2026-10-29 22:14:38.739900993 +0000 UTC m=+11.815218376" Oct 29 22:14:39.373080 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:39.372999 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:39.373246 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:39.373184 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:39.593796 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:39.593763 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:39.670251 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:39.670169 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-84hqt" event={"ID":"e1daa8f6-7854-4c2a-b207-bd6947f3e030","Type":"ContainerStarted","Data":"e9636a701e255af1c7e3fe5bce963d2afe020e59139e96887f3b19f98bb8edbe"} Oct 29 22:14:39.711505 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:39.711426 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-84hqt" podStartSLOduration=2.916096024 podStartE2EDuration="11.711405005s" totalImagesPullingTime="8.795308981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:39.710729546 +0000 UTC m=+12.786046857" watchObservedRunningTime="2026-10-29 22:14:39.711405005 +0000 UTC m=+12.786722318" Oct 29 22:14:39.752709 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:39.752677 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:40.372994 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.372906 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:40.373165 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:40.373072 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:40.373165 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.373106 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:40.373285 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:40.373227 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:40.382119 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.382089 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:40.534840 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.534800 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:40.673115 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.673039 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x8plf" event={"ID":"4bfd4dd9-a2a6-4ecd-9667-840738787716","Type":"ContainerStarted","Data":"cccd4aca82f23941a0cefcff7e812a23c4e8e620f6349d0b43f7e5eb6ff761bd"} Oct 29 22:14:40.676014 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.675987 2585 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="2443c0eec26179fd0e150f19de6466941e1d3843c44707a78276f59cb45fba4a" exitCode=0 Oct 29 22:14:40.676139 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.676091 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" event={"ID":"35dc91f0-4639-4294-a296-418c22a692c1","Type":"ContainerDied","Data":"2443c0eec26179fd0e150f19de6466941e1d3843c44707a78276f59cb45fba4a"} Oct 29 22:14:40.677898 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.677863 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:40.720659 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:40.720572 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x8plf" podStartSLOduration=1.431802972 podStartE2EDuration="12.720549522s" totalImagesPullingTime="11.28874655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:40.7205213 +0000 UTC m=+13.795838611" watchObservedRunningTime="2026-10-29 22:14:40.720549522 +0000 UTC m=+13.795866827" Oct 29 22:14:41.056069 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:41.055769 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:41.230040 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:41.230001 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:41.372463 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:41.372436 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:41.372610 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:41.372554 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:41.385687 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:41.385661 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:41.681499 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:41.681409 2585 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="630284d39864bebfffee3bae1b5eb2baebf84ef130d3fac3ee37ad720b797ab3" exitCode=0 Oct 29 22:14:41.681987 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:41.681488 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" event={"ID":"35dc91f0-4639-4294-a296-418c22a692c1","Type":"ContainerDied","Data":"630284d39864bebfffee3bae1b5eb2baebf84ef130d3fac3ee37ad720b797ab3"} Oct 29 22:14:41.880170 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:41.880135 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:42.373276 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:42.372791 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:42.373276 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:42.372919 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:42.373276 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:42.372960 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:42.373276 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:42.373058 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:42.809448 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:42.809348 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:42.976580 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:42.976538 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:43.145524 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:43.145449 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:43.299739 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:43.299690 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:43.373012 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:43.372972 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:43.373147 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:43.373106 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:43.581883 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:43.581851 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:43.689159 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:43.689123 2585 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="822dfeffeee52e2826b1bd98f7bbaec8cde5b9e77c6a02728c1544e330c63fa7" exitCode=0 Oct 29 22:14:43.689327 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:43.689187 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" event={"ID":"35dc91f0-4639-4294-a296-418c22a692c1","Type":"ContainerDied","Data":"822dfeffeee52e2826b1bd98f7bbaec8cde5b9e77c6a02728c1544e330c63fa7"} Oct 29 22:14:44.135914 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.135833 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:44.135914 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.135887 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:44.136350 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.135980 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:44.136350 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.136035 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:44.136350 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.136046 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret podName:ab618391-4607-4f00-ab02-819dc9897da1 nodeName:}" failed. No retries permitted until 2026-10-29 22:15:00.136030657 +0000 UTC m=+33.211347955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret") pod "global-pull-secret-syncer-5trph" (UID: "ab618391-4607-4f00-ab02-819dc9897da1") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:14:44.136350 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.136109 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs podName:b4a2ad5d-ac31-4ccf-8cd0-302df20209c0 nodeName:}" failed. No retries permitted until 2026-10-29 22:15:00.13609352 +0000 UTC m=+33.211410806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs") pod "network-metrics-daemon-nmlxs" (UID: "b4a2ad5d-ac31-4ccf-8cd0-302df20209c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:14:44.236948 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.236858 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:44.237088 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.236970 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:14:44.237088 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.236989 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:14:44.237088 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.237001 2585 projected.go:196] Error preparing data for projected volume kube-api-access-zn6ww for pod openshift-network-diagnostics/network-check-target-qq6r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:44.237088 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.237058 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww podName:1d3f4dde-95c6-41df-951b-316cd9d10f0a nodeName:}" failed. No retries permitted until 2026-10-29 22:15:00.237043334 +0000 UTC m=+33.312360637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zn6ww" (UniqueName: "kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww") pod "network-check-target-qq6r4" (UID: "1d3f4dde-95c6-41df-951b-316cd9d10f0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:14:44.372928 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.372893 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:44.372928 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.372917 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:44.373109 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.373013 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:44.373109 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:44.373089 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:44.380407 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.380359 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:44.570824 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.570738 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:44.697917 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.697890 2585 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="9964f29e5bbf6b00fdfe27a418ed875c0e595d88421958a4a1b1cf889ae6b52e" exitCode=0 Oct 29 22:14:44.698017 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.697930 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" event={"ID":"35dc91f0-4639-4294-a296-418c22a692c1","Type":"ContainerDied","Data":"9964f29e5bbf6b00fdfe27a418ed875c0e595d88421958a4a1b1cf889ae6b52e"} Oct 29 22:14:44.714693 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.714659 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:44.723813 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.723791 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lmmsc_d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79/dns-node-resolver/0.log" Oct 29 22:14:44.871678 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:44.871655 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:45.006771 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.006736 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:45.136645 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.136562 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:45.262515 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.262480 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:45.372972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.372939 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:45.373111 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:45.373046 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:45.702784 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.702745 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"007cb67e70c34b8156a2e6c6501c0b818407a84034c2d5a219a759aed554bbe0"} Oct 29 22:14:45.702988 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.702831 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"c5b6abfd70e5b05b0e7296e098c52405353819dd206bb8e14a2687074e7b0220"} Oct 29 22:14:45.702988 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.702874 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"1bfd3213321f39895912da93f65dec1c4d0e837fa61cbde68636152f2793e4cd"} Oct 29 22:14:45.702988 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.702931 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"97cd1382baf8187375112c576451268d4778c5a3b2ac156ce61f477f88bd8bf2"} Oct 29 22:14:45.702988 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.702972 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"8acada660dba3b3e390cd51cde8ed5f1c7638723aa7049c75b5461a7481b72f0"} Oct 29 22:14:45.703124 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.703014 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"bc580842c53cb35ad110b64a4e92accd82a565ece120880f19d8e5115039ac0f"} Oct 29 22:14:45.705877 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.705854 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" event={"ID":"35dc91f0-4639-4294-a296-418c22a692c1","Type":"ContainerStarted","Data":"71ef0a847bd0f242cf113985cd878d290aa8af20e2b439d6e8eee23eeb82abc3"} Oct 29 22:14:45.760761 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.760709 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dkbgr" podStartSLOduration=6.494362014 podStartE2EDuration="17.760695316s" totalImagesPullingTime="11.266333302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:45.747224405 +0000 UTC m=+18.822541713" watchObservedRunningTime="2026-10-29 22:14:45.760695316 +0000 UTC m=+18.836012624" Oct 29 22:14:45.923103 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:45.923076 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vhtpp_3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09/node-ca/0.log" Oct 29 22:14:46.369801 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:46.369768 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:46.372822 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:46.372799 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:46.373211 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:46.373182 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:46.373608 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:46.373592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:46.373769 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:46.373750 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:46.850626 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:46.850550 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:46.977973 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:46.977940 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:47.171304 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:47.171214 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:47.372997 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:47.372957 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:47.373639 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:47.373046 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:47.712888 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:47.712860 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"7abda4702900cfa37c84da66f9b7ab90b95ab660a9e786ab9009e35238566ca5"} Oct 29 22:14:48.036657 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:48.036579 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:48.171744 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:48.171708 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:48.370020 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:48.369931 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:48.372944 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:48.372927 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:48.373037 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:48.373021 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:48.373352 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:48.373075 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:48.373352 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:48.373197 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:48.571218 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:48.571182 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:48.772746 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:48.772653 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:48.967195 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:48.967166 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:49.169631 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:49.169600 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:49.372922 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:49.372602 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:49.373055 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:49.372962 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:49.721169 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:49.721088 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" event={"ID":"629cfb60-8afd-46e7-86a8-1de336684f20","Type":"ContainerStarted","Data":"3e56652ad6d9f9ab9e17c72001c914a47537752f88a23067c1c2905b87c97c18"} Oct 29 22:14:49.721452 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:49.721424 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:49.738020 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:49.737998 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:49.807087 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:49.807038 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" podStartSLOduration=5.850787845 podStartE2EDuration="21.807025202s" totalImagesPullingTime="15.956237357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:49.806602102 +0000 UTC m=+22.881919410" watchObservedRunningTime="2026-10-29 22:14:49.807025202 +0000 UTC m=+22.882342507" Oct 29 22:14:50.173988 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.173909 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:50.372419 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.372366 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:50.372573 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.372366 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:50.372573 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:50.372488 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:50.372681 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:50.372583 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:50.577682 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.577604 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:50.723561 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.723534 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:50.723724 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.723595 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:50.737933 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.737911 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:14:50.789950 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.789913 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:50.989544 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:50.989468 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:51.371033 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:51.370951 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:51.372968 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:51.372950 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:51.373061 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:51.373045 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:52.176171 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:52.176137 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:52.372709 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:52.372678 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:52.372709 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:52.372709 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:52.372886 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:52.372781 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:52.372926 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:52.372900 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:52.516258 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:52.516187 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:52.659496 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:52.659462 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:52.796751 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:52.796642 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:52.926206 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:52.926173 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:53.063341 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:53.063256 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:53.193668 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:53.193638 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:53.373040 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:53.372957 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:53.373204 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:53.373086 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:53.568743 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:53.568709 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:54.372146 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:54.372115 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:54.372634 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:54.372115 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:54.372634 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:54.372211 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:54.372634 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:54.372286 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:55.248240 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:55.248208 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:55.248451 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:55.248336 2585 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:14:55.248711 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:55.248697 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:14:55.372078 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:55.372047 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:55.372263 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:55.372179 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:56.373094 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:56.373010 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:56.373094 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:56.373039 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:56.373585 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:56.373120 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:56.373585 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:56.373225 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:57.373231 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:57.373196 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:57.373751 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:57.373294 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:57.685861 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:57.685783 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:57.822595 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:57.822564 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:57.953001 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:57.952922 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:58.085143 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:58.085106 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:58.214407 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:58.214321 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:58.372154 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:58.372121 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:14:58.372321 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:58.372125 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:14:58.372321 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:58.372222 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:14:58.372425 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:58.372335 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:14:59.372977 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:59.372940 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:14:59.373523 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:14:59.373052 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:14:59.637723 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:59.637641 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:14:59.767355 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:14:59.767322 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:00.153721 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:00.153686 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:00.153880 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:00.153728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:00.153880 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.153841 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 29 22:15:00.153971 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.153902 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret podName:ab618391-4607-4f00-ab02-819dc9897da1 nodeName:}" failed. No retries permitted until 2026-10-29 22:15:32.153888559 +0000 UTC m=+65.229205858 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret") pod "global-pull-secret-syncer-5trph" (UID: "ab618391-4607-4f00-ab02-819dc9897da1") : object "kube-system"/"original-pull-secret" not registered Oct 29 22:15:00.153971 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.153841 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:15:00.153971 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.153970 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs podName:b4a2ad5d-ac31-4ccf-8cd0-302df20209c0 nodeName:}" failed. No retries permitted until 2026-10-29 22:15:32.153958354 +0000 UTC m=+65.229275654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs") pod "network-metrics-daemon-nmlxs" (UID: "b4a2ad5d-ac31-4ccf-8cd0-302df20209c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 29 22:15:00.254855 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:00.254818 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:00.255030 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.254983 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 29 22:15:00.255030 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.255003 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 29 22:15:00.255030 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.255012 2585 projected.go:196] Error preparing data for projected volume kube-api-access-zn6ww for pod openshift-network-diagnostics/network-check-target-qq6r4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:15:00.255142 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.255062 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww podName:1d3f4dde-95c6-41df-951b-316cd9d10f0a nodeName:}" failed. No retries permitted until 2026-10-29 22:15:32.255050086 +0000 UTC m=+65.330367373 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zn6ww" (UniqueName: "kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww") pod "network-check-target-qq6r4" (UID: "1d3f4dde-95c6-41df-951b-316cd9d10f0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 29 22:15:00.372243 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:00.372207 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:00.372243 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:00.372235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:00.372489 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.372323 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:15:00.372489 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:00.372458 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:15:01.370368 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:01.370335 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:01.372436 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:01.372416 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:01.372539 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:01.372516 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:15:01.567717 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:01.567683 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:01.696453 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:01.696369 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:02.222910 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:02.222677 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:02.235362 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:02.235334 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qq6r4"] Oct 29 22:15:02.235511 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:02.235467 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:02.235597 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:02.235573 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:15:02.235772 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:02.235750 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5trph"] Oct 29 22:15:02.236015 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:02.236000 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:02.236218 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:02.236197 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:15:02.236533 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:02.236516 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmlxs"] Oct 29 22:15:02.236643 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:02.236600 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:02.236734 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:02.236691 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:15:03.375608 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:03.375582 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:03.375969 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:03.375680 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:15:04.372050 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:04.372018 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:04.372219 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:04.372019 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:04.372219 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:04.372126 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:15:04.372219 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:04.372211 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:15:05.001347 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:05.001314 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:05.129144 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:05.129065 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:05.313698 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:05.313662 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:05.376096 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:05.376071 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:05.376241 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:05.376167 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmlxs" podUID="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" Oct 29 22:15:06.372604 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.372576 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:06.372979 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.372575 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:06.372979 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:06.372678 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5trph" podUID="ab618391-4607-4f00-ab02-819dc9897da1" Oct 29 22:15:06.372979 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:06.372738 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq6r4" podUID="1d3f4dde-95c6-41df-951b-316cd9d10f0a" Oct 29 22:15:06.710425 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.710384 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeReady" Oct 29 22:15:06.710586 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.710528 2585 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 29 22:15:06.766640 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.766568 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6sljl"] Oct 29 22:15:06.797731 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.797703 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2vcms"] Oct 29 22:15:06.797880 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.797804 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.804013 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.803988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f585004-5478-4841-af5c-81c1c4bd9072-metrics-tls\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.804127 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.804041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnncl\" (UniqueName: \"kubernetes.io/projected/0f585004-5478-4841-af5c-81c1c4bd9072-kube-api-access-jnncl\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.804127 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.804078 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f585004-5478-4841-af5c-81c1c4bd9072-config-volume\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.804127 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.804096 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f585004-5478-4841-af5c-81c1c4bd9072-tmp-dir\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.810409 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.810370 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-446fn"] Oct 29 22:15:06.810685 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.810488 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:06.813342 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.813321 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:15:06.813342 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.813338 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qnnrl\"" Oct 29 22:15:06.813513 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.813414 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:15:06.824919 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.824898 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vcms"] Oct 29 22:15:06.825005 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.824929 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6sljl"] Oct 29 22:15:06.825005 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.824952 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-446fn"] Oct 29 22:15:06.825005 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.824998 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:06.825724 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.825705 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:15:06.825855 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.825838 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:15:06.825912 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.825878 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:15:06.825953 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.825929 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h6g72\"" Oct 29 22:15:06.905358 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905290 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dtsh\" (UniqueName: \"kubernetes.io/projected/3b731a32-138f-4eb8-830a-7078d3667e68-kube-api-access-6dtsh\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:06.905358 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905317 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a06ed94e-63bb-4a18-b959-4175a0972ffb-crio-socket\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:06.905358 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905337 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a06ed94e-63bb-4a18-b959-4175a0972ffb-data-volume\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:06.905358 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905351 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88ts\" (UniqueName: \"kubernetes.io/projected/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-api-access-k88ts\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:06.905585 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905453 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b731a32-138f-4eb8-830a-7078d3667e68-cert\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:06.905585 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905488 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a06ed94e-63bb-4a18-b959-4175a0972ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:06.905654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f585004-5478-4841-af5c-81c1c4bd9072-metrics-tls\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.905654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905601 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnncl\" (UniqueName: \"kubernetes.io/projected/0f585004-5478-4841-af5c-81c1c4bd9072-kube-api-access-jnncl\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.905654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905620 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f585004-5478-4841-af5c-81c1c4bd9072-config-volume\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.905654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905637 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f585004-5478-4841-af5c-81c1c4bd9072-tmp-dir\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.905654 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.905652 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:06.906025 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.906009 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f585004-5478-4841-af5c-81c1c4bd9072-tmp-dir\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.906102 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.906088 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f585004-5478-4841-af5c-81c1c4bd9072-config-volume\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.909401 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.909375 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f585004-5478-4841-af5c-81c1c4bd9072-metrics-tls\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:06.913603 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.913584 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:15:06.913738 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.913720 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:15:06.913882 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.913866 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:15:06.913941 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.913884 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r9x6m\"" Oct 29 22:15:06.913941 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.913891 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:15:06.924723 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:06.924703 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnncl\" (UniqueName: \"kubernetes.io/projected/0f585004-5478-4841-af5c-81c1c4bd9072-kube-api-access-jnncl\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:07.006857 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.006823 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.007043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.006877 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtsh\" (UniqueName: \"kubernetes.io/projected/3b731a32-138f-4eb8-830a-7078d3667e68-kube-api-access-6dtsh\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:07.007043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.006895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a06ed94e-63bb-4a18-b959-4175a0972ffb-crio-socket\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.007043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.006921 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a06ed94e-63bb-4a18-b959-4175a0972ffb-data-volume\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.007181 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.007037 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88ts\" (UniqueName: \"kubernetes.io/projected/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-api-access-k88ts\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.007181 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.007078 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b731a32-138f-4eb8-830a-7078d3667e68-cert\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:07.007181 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.007105 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a06ed94e-63bb-4a18-b959-4175a0972ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.007181 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.007146 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a06ed94e-63bb-4a18-b959-4175a0972ffb-crio-socket\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.007308 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.007262 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a06ed94e-63bb-4a18-b959-4175a0972ffb-data-volume\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.007500 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.007478 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.009377 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.009354 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b731a32-138f-4eb8-830a-7078d3667e68-cert\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:07.009463 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.009359 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a06ed94e-63bb-4a18-b959-4175a0972ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.026566 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.026547 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88ts\" (UniqueName: \"kubernetes.io/projected/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-api-access-k88ts\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.026765 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.026750 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtsh\" (UniqueName: \"kubernetes.io/projected/3b731a32-138f-4eb8-830a-7078d3667e68-kube-api-access-6dtsh\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:07.059382 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.059357 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:07.131043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.131011 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:07.141141 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.141121 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:07.189921 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.189897 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:07.376493 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.376468 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:07.391972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.391950 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:15:07.392085 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.392014 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbfws\"" Oct 29 22:15:07.509770 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.509695 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6sljl"] Oct 29 22:15:07.511664 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:07.511641 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f585004_5478_4841_af5c_81c1c4bd9072.slice/crio-26b299caf779a2a40922c91faf822823b3c1d619edb455491ac93f506c234cf0 WatchSource:0}: Error finding container 26b299caf779a2a40922c91faf822823b3c1d619edb455491ac93f506c234cf0: Status 404 returned error can't find the container with id 26b299caf779a2a40922c91faf822823b3c1d619edb455491ac93f506c234cf0 Oct 29 22:15:07.519495 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.519473 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vcms"] Oct 29 22:15:07.520631 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:07.520608 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b731a32_138f_4eb8_830a_7078d3667e68.slice/crio-f6b9149c5d7276214044a17e1af22c9e6935502e58ad3e74c46ae8b1cec067a5 WatchSource:0}: Error finding container f6b9149c5d7276214044a17e1af22c9e6935502e58ad3e74c46ae8b1cec067a5: Status 404 returned error can't find the container with id f6b9149c5d7276214044a17e1af22c9e6935502e58ad3e74c46ae8b1cec067a5 Oct 29 22:15:07.522567 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.522434 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-446fn"] Oct 29 22:15:07.523556 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:07.523474 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06ed94e_63bb_4a18_b959_4175a0972ffb.slice/crio-52c3a9dc13a1dba0fd34874033ba5a08d028a086027458a96c674fae39bd1ea3 WatchSource:0}: Error finding container 52c3a9dc13a1dba0fd34874033ba5a08d028a086027458a96c674fae39bd1ea3: Status 404 returned error can't find the container with id 52c3a9dc13a1dba0fd34874033ba5a08d028a086027458a96c674fae39bd1ea3 Oct 29 22:15:07.668374 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.668195 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:07.758075 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.758043 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-446fn" event={"ID":"a06ed94e-63bb-4a18-b959-4175a0972ffb","Type":"ContainerStarted","Data":"f756cabaea056c52355b43126eb16c4e199375ba8a59c46ae3fdef9945956c73"} Oct 29 22:15:07.758304 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.758102 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-446fn" event={"ID":"a06ed94e-63bb-4a18-b959-4175a0972ffb","Type":"ContainerStarted","Data":"52c3a9dc13a1dba0fd34874033ba5a08d028a086027458a96c674fae39bd1ea3"} Oct 29 22:15:07.759117 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.759097 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vcms" event={"ID":"3b731a32-138f-4eb8-830a-7078d3667e68","Type":"ContainerStarted","Data":"f6b9149c5d7276214044a17e1af22c9e6935502e58ad3e74c46ae8b1cec067a5"} Oct 29 22:15:07.760127 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.760107 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6sljl" event={"ID":"0f585004-5478-4841-af5c-81c1c4bd9072","Type":"ContainerStarted","Data":"26b299caf779a2a40922c91faf822823b3c1d619edb455491ac93f506c234cf0"} Oct 29 22:15:07.819616 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:07.819584 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:08.157677 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.157593 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:08.372938 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.372909 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:08.373720 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.373280 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:08.388847 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.388805 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4fh5h\"" Oct 29 22:15:08.389171 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.388860 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:15:08.389171 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.388854 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:15:08.401956 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.401934 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:15:08.402065 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.401999 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dk2rc\"" Oct 29 22:15:08.765532 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:08.765493 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-446fn" event={"ID":"a06ed94e-63bb-4a18-b959-4175a0972ffb","Type":"ContainerStarted","Data":"b729e058e3814d5742c92b6c655e71e672abb7e9c988959a3fc7157fcbdba416"} Oct 29 22:15:09.837807 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:09.837767 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:09.971596 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:09.971556 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:10.440450 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.440415 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:10.584916 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.584679 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:10.649874 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.649837 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc"] Oct 29 22:15:10.652353 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.652336 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.667526 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.667503 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 29 22:15:10.667705 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.667688 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:15:10.680982 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.680762 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:15:10.680982 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.680834 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hg8d9\"" Oct 29 22:15:10.680982 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.680883 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:15:10.680982 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.680954 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:15:10.681702 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.681682 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc"] Oct 29 22:15:10.694823 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.694799 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-424dq"] Oct 29 22:15:10.696765 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.696751 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.702683 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.702233 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk"] Oct 29 22:15:10.704495 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.704479 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.714233 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.714216 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:15:10.714317 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.714240 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:15:10.714362 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.714349 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28glh\"" Oct 29 22:15:10.731138 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.731113 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk"] Oct 29 22:15:10.738319 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738290 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.738462 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738330 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-textfile\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.738515 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738461 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-metrics-client-ca\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.738515 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738497 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.738587 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738530 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.738587 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738551 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-sys\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.738678 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738666 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.738761 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738746 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-root\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.738835 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738767 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-wtmp\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.738835 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738816 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-accelerators-collector-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.738942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738859 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.738942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738910 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.738942 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-tls\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.739041 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738967 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqh2\" (UniqueName: \"kubernetes.io/projected/3f6e07a6-8957-44f5-9203-425259d84060-kube-api-access-mwqh2\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.739041 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.738996 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71813e11-80fa-4c04-ba56-cd1bc57d3f35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.739223 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.739077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.739223 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.739129 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb77h\" (UniqueName: \"kubernetes.io/projected/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-kube-api-access-cb77h\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.739223 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.739161 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfcc9\" (UniqueName: \"kubernetes.io/projected/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-api-access-tfcc9\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.739341 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.739228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.741162 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.741136 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:10.748604 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.748583 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:15:10.748740 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.748724 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 29 22:15:10.748828 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.748812 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-656vh\"" Oct 29 22:15:10.768222 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.768196 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:15:10.768320 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.768279 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 29 22:15:10.771788 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.771760 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-446fn" event={"ID":"a06ed94e-63bb-4a18-b959-4175a0972ffb","Type":"ContainerStarted","Data":"aca00c5dd56dd543a3dcd121a0493bb00855d1f852d1789aff68f9f6653f6cc2"} Oct 29 22:15:10.773110 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.773093 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vcms" event={"ID":"3b731a32-138f-4eb8-830a-7078d3667e68","Type":"ContainerStarted","Data":"50de9f3ef8cb7b0ad5a1974c93bdaec6cd38766f9035df0788d951652f522ad0"} Oct 29 22:15:10.774733 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.774710 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6sljl" event={"ID":"0f585004-5478-4841-af5c-81c1c4bd9072","Type":"ContainerStarted","Data":"914882d58c7d06c12a4cf24a0400ebe84974ca1a97949cc241e63854de35166a"} Oct 29 22:15:10.774806 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.774749 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6sljl" event={"ID":"0f585004-5478-4841-af5c-81c1c4bd9072","Type":"ContainerStarted","Data":"dd125a116341e8ee6846e859a49088d7716687b584a8d2db1d425414a873c40f"} Oct 29 22:15:10.774806 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.774774 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:10.819136 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.819091 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2vcms" podStartSLOduration=1.889076543 podStartE2EDuration="4.819075175s" totalImagesPullingTime="2.929998632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:10.818432975 +0000 UTC m=+43.893750280" watchObservedRunningTime="2026-10-29 22:15:10.819075175 +0000 UTC m=+43.894392514" Oct 29 22:15:10.839643 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.839549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.839643 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.839591 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.839643 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.839615 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-tls\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.840191 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.839643 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqh2\" (UniqueName: \"kubernetes.io/projected/3f6e07a6-8957-44f5-9203-425259d84060-kube-api-access-mwqh2\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.840191 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.839670 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71813e11-80fa-4c04-ba56-cd1bc57d3f35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.840191 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.839693 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.840191 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.839977 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb77h\" (UniqueName: \"kubernetes.io/projected/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-kube-api-access-cb77h\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.840191 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840017 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfcc9\" (UniqueName: \"kubernetes.io/projected/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-api-access-tfcc9\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.840191 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840046 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.840191 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.840191 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840144 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-textfile\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840198 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-metrics-client-ca\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840325 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-sys\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840381 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-root\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840448 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840469 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-wtmp\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-accelerators-collector-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.840562 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840546 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71813e11-80fa-4c04-ba56-cd1bc57d3f35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.840961 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.840750 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-textfile\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.841123 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.841000 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-wtmp\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.841729 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.841705 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-sys\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.842041 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.842015 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.842474 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.842324 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-accelerators-collector-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.842474 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.842413 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-root\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.843178 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.843136 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.843673 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.843652 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-metrics-client-ca\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.844812 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.844785 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.844914 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.844795 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-tls\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.844914 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.844829 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.844914 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.844879 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.844914 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.844876 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.845171 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.845152 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.861588 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.861338 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-446fn" podStartSLOduration=2.140544297 podStartE2EDuration="4.861326713s" totalImagesPullingTime="2.720782416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:15:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:10.860553166 +0000 UTC m=+43.935870474" watchObservedRunningTime="2026-10-29 22:15:10.861326713 +0000 UTC m=+43.936644017" Oct 29 22:15:10.865349 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.865328 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqh2\" (UniqueName: \"kubernetes.io/projected/3f6e07a6-8957-44f5-9203-425259d84060-kube-api-access-mwqh2\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:10.865349 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.865339 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfcc9\" (UniqueName: \"kubernetes.io/projected/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-api-access-tfcc9\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:10.865531 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.865471 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb77h\" (UniqueName: \"kubernetes.io/projected/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-kube-api-access-cb77h\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:10.882338 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.882307 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:10.901142 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.901096 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-6sljl" podStartSLOduration=2.737724345 podStartE2EDuration="4.901065645s" totalImagesPullingTime="2.1633413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:10.900563122 +0000 UTC m=+43.975880429" watchObservedRunningTime="2026-10-29 22:15:10.901065645 +0000 UTC m=+43.976382950" Oct 29 22:15:10.964934 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:10.964900 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:11.010281 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.010240 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:11.016647 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.016616 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:11.021779 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:11.021748 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6e07a6_8957_44f5_9203_425259d84060.slice/crio-54828e6587593c8fe349bb25a9f11d49ea0da33a7e857275255c5ec5170c972a WatchSource:0}: Error finding container 54828e6587593c8fe349bb25a9f11d49ea0da33a7e857275255c5ec5170c972a: Status 404 returned error can't find the container with id 54828e6587593c8fe349bb25a9f11d49ea0da33a7e857275255c5ec5170c972a Oct 29 22:15:11.053624 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.053594 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:11.245186 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.245006 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:11.272498 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.272462 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc"] Oct 29 22:15:11.273293 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:11.273267 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c71f14d_b3bf_4942_beaa_6cdf18e331b9.slice/crio-c2c1fb0200c80168207b2c84d2d237d12c86dc4d8773b20cd5cfa5566552055e WatchSource:0}: Error finding container c2c1fb0200c80168207b2c84d2d237d12c86dc4d8773b20cd5cfa5566552055e: Status 404 returned error can't find the container with id c2c1fb0200c80168207b2c84d2d237d12c86dc4d8773b20cd5cfa5566552055e Oct 29 22:15:11.310797 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.310772 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk"] Oct 29 22:15:11.312033 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:11.312010 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71813e11_80fa_4c04_ba56_cd1bc57d3f35.slice/crio-6d845444a56b8edc586e7039acc0ab3acfa7ff6c2fb1b73ea5754089bfc0f63c WatchSource:0}: Error finding container 6d845444a56b8edc586e7039acc0ab3acfa7ff6c2fb1b73ea5754089bfc0f63c: Status 404 returned error can't find the container with id 6d845444a56b8edc586e7039acc0ab3acfa7ff6c2fb1b73ea5754089bfc0f63c Oct 29 22:15:11.780203 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.780162 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-424dq" event={"ID":"3f6e07a6-8957-44f5-9203-425259d84060","Type":"ContainerStarted","Data":"54828e6587593c8fe349bb25a9f11d49ea0da33a7e857275255c5ec5170c972a"} Oct 29 22:15:11.782650 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.782522 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" event={"ID":"9c71f14d-b3bf-4942-beaa-6cdf18e331b9","Type":"ContainerStarted","Data":"1b9c2c5dbdbe71699ab06816d2bd76a6ccce24601590403e85c9483609992450"} Oct 29 22:15:11.782650 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.782583 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" event={"ID":"9c71f14d-b3bf-4942-beaa-6cdf18e331b9","Type":"ContainerStarted","Data":"5903517e1a2276b44690ad4f585b14a274c0939d04ebbcf9f624a90ebc4d9e9a"} Oct 29 22:15:11.782650 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.782608 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" event={"ID":"9c71f14d-b3bf-4942-beaa-6cdf18e331b9","Type":"ContainerStarted","Data":"c2c1fb0200c80168207b2c84d2d237d12c86dc4d8773b20cd5cfa5566552055e"} Oct 29 22:15:11.783755 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.783665 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" event={"ID":"71813e11-80fa-4c04-ba56-cd1bc57d3f35","Type":"ContainerStarted","Data":"6d845444a56b8edc586e7039acc0ab3acfa7ff6c2fb1b73ea5754089bfc0f63c"} Oct 29 22:15:11.905633 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:11.905606 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:12.053126 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.053029 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:12.189386 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.189349 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:12.633291 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.633261 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:12.769570 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.769354 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:12.794459 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.794411 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" event={"ID":"71813e11-80fa-4c04-ba56-cd1bc57d3f35","Type":"ContainerStarted","Data":"db0c128e33335e5f42c766fadf4a2e5ab3aa4134c2973ac3a9120e261f209dcc"} Oct 29 22:15:12.794657 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.794504 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" event={"ID":"71813e11-80fa-4c04-ba56-cd1bc57d3f35","Type":"ContainerStarted","Data":"63150a9b1d7997909d29b74c9c5f03503f8a704c321bf29d9a6f02de7265969a"} Oct 29 22:15:12.794657 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.794533 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" event={"ID":"71813e11-80fa-4c04-ba56-cd1bc57d3f35","Type":"ContainerStarted","Data":"e5891bc8fb0d87bb8afdcce7f55cf675f688e9b4e1710eca19bba3cc750693e4"} Oct 29 22:15:12.796086 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.796061 2585 generic.go:356] "Generic (PLEG): container finished" podID="3f6e07a6-8957-44f5-9203-425259d84060" containerID="444db050ab90c2f66bc2da43992f7cbb38a7de80f920ffb3eb18d4586d3c09a8" exitCode=0 Oct 29 22:15:12.796204 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.796151 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-424dq" event={"ID":"3f6e07a6-8957-44f5-9203-425259d84060","Type":"ContainerDied","Data":"444db050ab90c2f66bc2da43992f7cbb38a7de80f920ffb3eb18d4586d3c09a8"} Oct 29 22:15:12.798495 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.798468 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" event={"ID":"9c71f14d-b3bf-4942-beaa-6cdf18e331b9","Type":"ContainerStarted","Data":"b77f98305f99cb8ff6ac35bdc2cc46371ee704395f296be8d80e308b444fb3c3"} Oct 29 22:15:12.861556 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.861505 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" podStartSLOduration=1.668399763 podStartE2EDuration="2.861491711s" totalImagesPullingTime="1.193091948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:12.848144838 +0000 UTC m=+45.923462148" watchObservedRunningTime="2026-10-29 22:15:12.861491711 +0000 UTC m=+45.936809019" Oct 29 22:15:12.931936 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.931883 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" podStartSLOduration=1.8977284129999998 podStartE2EDuration="2.931865511s" totalImagesPullingTime="1.034137098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:12.930767852 +0000 UTC m=+46.006085162" watchObservedRunningTime="2026-10-29 22:15:12.931865511 +0000 UTC m=+46.007182821" Oct 29 22:15:12.951951 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.951848 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-848cccd68d-hlbqr"] Oct 29 22:15:12.954763 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.954744 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:12.972517 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.972433 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 29 22:15:12.972517 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.972490 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 29 22:15:12.972517 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.972490 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 29 22:15:12.972734 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.972490 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 29 22:15:12.972734 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.972538 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 29 22:15:12.972734 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.972597 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t5dlz\"" Oct 29 22:15:12.972734 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.972658 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-csh82vmno0a66\"" Oct 29 22:15:12.977653 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:12.977632 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-848cccd68d-hlbqr"] Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062139 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062219 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82b16e7a-7a16-409f-9411-d23156de1fe1-metrics-client-ca\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062287 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-grpc-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062328 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062375 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b16e7a-7a16-409f-9411-d23156de1fe1-tmp\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062433 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.064434 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.062506 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6r2d\" (UniqueName: \"kubernetes.io/projected/82b16e7a-7a16-409f-9411-d23156de1fe1-kube-api-access-g6r2d\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.074541 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.074517 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:13.162978 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.162943 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163126 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.162983 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163126 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.163005 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b16e7a-7a16-409f-9411-d23156de1fe1-tmp\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163126 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.163028 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163126 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.163071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6r2d\" (UniqueName: \"kubernetes.io/projected/82b16e7a-7a16-409f-9411-d23156de1fe1-kube-api-access-g6r2d\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163126 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.163097 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163351 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.163289 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163454 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.163433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82b16e7a-7a16-409f-9411-d23156de1fe1-metrics-client-ca\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163521 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.163448 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b16e7a-7a16-409f-9411-d23156de1fe1-tmp\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.163521 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.163484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-grpc-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.164092 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.164066 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82b16e7a-7a16-409f-9411-d23156de1fe1-metrics-client-ca\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.167321 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.167296 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.167459 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.167305 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.167459 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.167382 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.167563 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.167506 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-grpc-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.167563 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.167554 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.167660 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.167641 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.182479 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.182456 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6r2d\" (UniqueName: \"kubernetes.io/projected/82b16e7a-7a16-409f-9411-d23156de1fe1-kube-api-access-g6r2d\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.204134 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.204107 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:13.270481 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.270451 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:13.585428 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.585388 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-848cccd68d-hlbqr"] Oct 29 22:15:13.586851 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:13.586826 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b16e7a_7a16_409f_9411_d23156de1fe1.slice/crio-ca593fa5fc933d17b00aa85fa018eefe914c1acd4923f704d4174a1635ef70af WatchSource:0}: Error finding container ca593fa5fc933d17b00aa85fa018eefe914c1acd4923f704d4174a1635ef70af: Status 404 returned error can't find the container with id ca593fa5fc933d17b00aa85fa018eefe914c1acd4923f704d4174a1635ef70af Oct 29 22:15:13.802719 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.802691 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" event={"ID":"82b16e7a-7a16-409f-9411-d23156de1fe1","Type":"ContainerStarted","Data":"ca593fa5fc933d17b00aa85fa018eefe914c1acd4923f704d4174a1635ef70af"} Oct 29 22:15:13.804747 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.804712 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-424dq" event={"ID":"3f6e07a6-8957-44f5-9203-425259d84060","Type":"ContainerStarted","Data":"10c0fcf98128318f040c54c2c0772c5fc5115237ac2960ce4512cad4e9d82a07"} Oct 29 22:15:13.804883 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.804764 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-424dq" event={"ID":"3f6e07a6-8957-44f5-9203-425259d84060","Type":"ContainerStarted","Data":"3bd8eabab5601b0fb029c4baccca1062532c70bf180c6f53618407e15390470a"} Oct 29 22:15:13.849831 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.849719 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-424dq" podStartSLOduration=2.961108985 podStartE2EDuration="3.849706139s" totalImagesPullingTime="803.028873ms" totalInitContainerRuntime="85.568281ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:13.849078795 +0000 UTC m=+46.924396103" watchObservedRunningTime="2026-10-29 22:15:13.849706139 +0000 UTC m=+46.925023463" Oct 29 22:15:13.952908 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:13.952872 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:14.088843 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:14.088813 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:14.299261 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:14.299215 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:14.936321 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:14.936288 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:15.074759 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.074723 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:15.520814 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.520793 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs"] Oct 29 22:15:15.522550 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.522532 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:15.531368 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.531346 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:15.537559 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.537542 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 29 22:15:15.537639 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.537612 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tqdrf\"" Oct 29 22:15:15.552278 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.552250 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs"] Oct 29 22:15:15.583113 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.583093 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:15.583113 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.583121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn28n\" (UniqueName: \"kubernetes.io/projected/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-kube-api-access-wn28n\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:15.683847 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.683757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:15.683847 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.683800 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn28n\" (UniqueName: \"kubernetes.io/projected/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-kube-api-access-wn28n\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:15.684065 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:15.683942 2585 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Oct 29 22:15:15.684065 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:15.684021 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert podName:d0c9a67d-21dc-44cf-ab1f-1ef20fba70af nodeName:}" failed. No retries permitted until 2026-10-29 22:15:16.183998152 +0000 UTC m=+49.259315458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert") pod "monitoring-plugin-77f4f76594-vn7xs" (UID: "d0c9a67d-21dc-44cf-ab1f-1ef20fba70af") : secret "monitoring-plugin-cert" not found Oct 29 22:15:15.707015 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.706980 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn28n\" (UniqueName: \"kubernetes.io/projected/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-kube-api-access-wn28n\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:15.737346 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.737317 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:15.815070 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.815036 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" event={"ID":"82b16e7a-7a16-409f-9411-d23156de1fe1","Type":"ContainerStarted","Data":"acd7040cba31904e768d51f2abd1b34156d72cb06fc70d26c7e5c56a19ed0379"} Oct 29 22:15:15.815201 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.815104 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" event={"ID":"82b16e7a-7a16-409f-9411-d23156de1fe1","Type":"ContainerStarted","Data":"d33876ad37808dc03d1489f9a0867a4d96df950182a1c0d527b9537809416b72"} Oct 29 22:15:15.815201 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.815129 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" event={"ID":"82b16e7a-7a16-409f-9411-d23156de1fe1","Type":"ContainerStarted","Data":"dc93ff07fbb372a9bf28086eaac4f8b5a80f7c9e342e50226d8bc659de0181b2"} Oct 29 22:15:15.937451 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:15.937375 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:16.188278 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.188196 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:16.190735 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.190715 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:16.433006 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.432985 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:16.727489 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.727448 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs"] Oct 29 22:15:16.728310 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:16.728289 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c9a67d_21dc_44cf_ab1f_1ef20fba70af.slice/crio-8432b647b5c4d3a0af1525c48fc1e0420a6dcc38532b8827781a9c9be3ec3b03 WatchSource:0}: Error finding container 8432b647b5c4d3a0af1525c48fc1e0420a6dcc38532b8827781a9c9be3ec3b03: Status 404 returned error can't find the container with id 8432b647b5c4d3a0af1525c48fc1e0420a6dcc38532b8827781a9c9be3ec3b03 Oct 29 22:15:16.821175 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.821140 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" event={"ID":"82b16e7a-7a16-409f-9411-d23156de1fe1","Type":"ContainerStarted","Data":"08924b3dcc688e848a93696ba6cb5d844c5c92a8d5ebdb5f8d2770f4f0a44e19"} Oct 29 22:15:16.821325 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.821222 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" event={"ID":"82b16e7a-7a16-409f-9411-d23156de1fe1","Type":"ContainerStarted","Data":"8080216e14982c48cd9d3a5036b16645b6815dff43eb3c2c51652c87aa2055a5"} Oct 29 22:15:16.821325 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.821272 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:16.821325 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.821314 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" event={"ID":"82b16e7a-7a16-409f-9411-d23156de1fe1","Type":"ContainerStarted","Data":"fb6753220343e3bc9b797e32c08098a3b9102c4bcbfcc1466d810c2e43a668dc"} Oct 29 22:15:16.822320 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.822299 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" event={"ID":"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af","Type":"ContainerStarted","Data":"8432b647b5c4d3a0af1525c48fc1e0420a6dcc38532b8827781a9c9be3ec3b03"} Oct 29 22:15:16.891202 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:16.891146 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" podStartSLOduration=2.360914957 podStartE2EDuration="4.891133097s" totalImagesPullingTime="2.53021814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:16.877941603 +0000 UTC m=+49.953258913" watchObservedRunningTime="2026-10-29 22:15:16.891133097 +0000 UTC m=+49.966450403" Oct 29 22:15:17.037460 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.037369 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:17.257700 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.257667 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:17.586266 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.586232 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:15:17.589322 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.589293 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.606305 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.606266 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 22:15:17.606571 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.606531 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 22:15:17.606571 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.606562 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 22:15:17.607096 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.607045 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 22:15:17.607238 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.607216 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 22:15:17.607337 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.607267 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 22:15:17.609240 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.608663 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kptl2\"" Oct 29 22:15:17.609240 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.608743 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7kpma9rsabkp1\"" Oct 29 22:15:17.622043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.621951 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 22:15:17.622043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.621992 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 22:15:17.622043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.621992 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 22:15:17.622043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.621991 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 22:15:17.622043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.622023 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 22:15:17.622345 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.622080 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 22:15:17.622345 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.622249 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 22:15:17.623162 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.623144 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:15:17.645888 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.645861 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 22:15:17.648608 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.648587 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 22:15:17.653721 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.653698 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:17.702964 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.702916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.702964 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.702963 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703183 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.702982 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27m6m\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703183 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703092 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703183 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703183 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703144 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703183 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703172 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703420 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703254 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703420 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703297 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703420 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703355 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703420 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703385 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703619 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703471 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703619 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703512 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703619 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703539 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703619 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703610 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703796 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703677 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703796 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703720 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703796 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703781 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703890 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.703890 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.703876 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.804795 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.804757 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.804972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.804804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.804972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.804844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.804972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.804872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.804972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.804895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.804972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.804917 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.804972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.804935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.804972 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.804959 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805000 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805016 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805097 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805110 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27m6m\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805152 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805201 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805247 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805267 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.805803 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.805736 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.806259 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.806236 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.806883 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.806845 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.806976 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.806952 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.807833 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.807806 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.808888 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.808860 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.808992 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.808971 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.809140 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.809103 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.809810 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.809436 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.809810 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.809765 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.810219 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.810180 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.810798 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.810772 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.810798 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.810788 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.810798 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.810797 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.811133 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.811114 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.811706 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.811661 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.812151 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.812133 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.812844 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.812757 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.812844 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.812810 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.828247 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.828217 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27m6m\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:17.909530 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:17.909457 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:18.237846 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:18.237821 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:15:18.238952 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:18.238927 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efed476_7ba2_4cce_9d12_eb2e1e457160.slice/crio-495cd647124ee7f1bb17ba294b19464e4ae192eeafa4b1623c98792cb046bcc4 WatchSource:0}: Error finding container 495cd647124ee7f1bb17ba294b19464e4ae192eeafa4b1623c98792cb046bcc4: Status 404 returned error can't find the container with id 495cd647124ee7f1bb17ba294b19464e4ae192eeafa4b1623c98792cb046bcc4 Oct 29 22:15:18.831103 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:18.831072 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" event={"ID":"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af","Type":"ContainerStarted","Data":"8a71f4a3e8418c7e1dc27cac3aaabcf543b3459e88f1803d2f9acaca06dd2622"} Oct 29 22:15:18.831576 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:18.831216 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:18.832401 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:18.832357 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerStarted","Data":"495cd647124ee7f1bb17ba294b19464e4ae192eeafa4b1623c98792cb046bcc4"} Oct 29 22:15:18.836179 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:18.836149 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:18.870402 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:18.870354 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" podStartSLOduration=2.568716404 podStartE2EDuration="3.870301703s" totalImagesPullingTime="1.301585299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:18.869839974 +0000 UTC m=+51.945157305" watchObservedRunningTime="2026-10-29 22:15:18.870301703 +0000 UTC m=+51.945619013" Oct 29 22:15:19.837665 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:19.837628 2585 generic.go:356] "Generic (PLEG): container finished" podID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" exitCode=0 Oct 29 22:15:19.838049 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:19.837712 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerDied","Data":"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436"} Oct 29 22:15:20.786843 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:20.786807 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:21.456284 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:21.456252 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:21.589119 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:21.589075 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:22.736560 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:22.736532 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:22.835372 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:22.835347 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:22.850456 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:22.850425 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerStarted","Data":"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100"} Oct 29 22:15:22.850552 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:22.850509 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerStarted","Data":"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e"} Oct 29 22:15:23.256214 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:23.256179 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:23.394012 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:23.393984 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:23.524914 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:23.524837 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:23.858447 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:23.858362 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerStarted","Data":"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627"} Oct 29 22:15:23.858819 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:23.858449 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerStarted","Data":"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81"} Oct 29 22:15:23.858819 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:23.858478 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerStarted","Data":"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995"} Oct 29 22:15:23.858819 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:23.858502 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerStarted","Data":"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672"} Oct 29 22:15:23.925526 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:23.925466 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.249677543 podStartE2EDuration="6.925449452s" totalImagesPullingTime="3.658644737s" totalInitContainerRuntime="17.127172ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:23.924274732 +0000 UTC m=+56.999592040" watchObservedRunningTime="2026-10-29 22:15:23.925449452 +0000 UTC m=+57.000766765" Oct 29 22:15:25.649165 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:25.649132 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:25.785794 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:25.785760 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:25.921342 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:25.921265 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:27.466007 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:27.465973 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:27.604752 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:27.604718 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:27.910512 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:27.910442 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:29.631336 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:29.631302 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:29.765043 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:29.765013 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:29.930328 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:29.930244 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:30.062967 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:30.062935 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:32.227069 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.227035 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:32.227069 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.227070 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:32.241699 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.241676 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:15:32.249837 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.249816 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:32.254743 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.254721 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:15:32.259706 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.259682 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:32.315959 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.315931 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbfws\"" Oct 29 22:15:32.321997 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.321978 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:32.328048 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.328029 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:32.342481 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.342459 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:15:32.362816 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.362788 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:15:32.372136 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.372108 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:32.415264 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.415239 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dk2rc\"" Oct 29 22:15:32.421449 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.421427 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:32.430488 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.430467 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4fh5h\"" Oct 29 22:15:32.436526 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.436508 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:32.615591 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.615563 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmlxs"] Oct 29 22:15:32.616451 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:32.616419 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a2ad5d_ac31_4ccf_8cd0_302df20209c0.slice/crio-d0f311b79812b4cda2233f92166438c744447afdc5ad8d81b2d6a58cef8f556c WatchSource:0}: Error finding container d0f311b79812b4cda2233f92166438c744447afdc5ad8d81b2d6a58cef8f556c: Status 404 returned error can't find the container with id d0f311b79812b4cda2233f92166438c744447afdc5ad8d81b2d6a58cef8f556c Oct 29 22:15:32.713907 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.713882 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5trph"] Oct 29 22:15:32.715010 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:32.714985 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab618391_4607_4f00_ab02_819dc9897da1.slice/crio-506e60b79a4bdebfc8d07980c3d51f71ca44f1554ceeaf16877b3f0825f595dd WatchSource:0}: Error finding container 506e60b79a4bdebfc8d07980c3d51f71ca44f1554ceeaf16877b3f0825f595dd: Status 404 returned error can't find the container with id 506e60b79a4bdebfc8d07980c3d51f71ca44f1554ceeaf16877b3f0825f595dd Oct 29 22:15:32.736855 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.736832 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qq6r4"] Oct 29 22:15:32.737405 ip-10-0-133-251 kubenswrapper[2585]: W1029 22:15:32.737375 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3f4dde_95c6_41df_951b_316cd9d10f0a.slice/crio-680af67bdbfa9cdff8d076092e42778e5527522ad7b901ea4701739b293f52f2 WatchSource:0}: Error finding container 680af67bdbfa9cdff8d076092e42778e5527522ad7b901ea4701739b293f52f2: Status 404 returned error can't find the container with id 680af67bdbfa9cdff8d076092e42778e5527522ad7b901ea4701739b293f52f2 Oct 29 22:15:32.890516 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.890428 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5trph" event={"ID":"ab618391-4607-4f00-ab02-819dc9897da1","Type":"ContainerStarted","Data":"506e60b79a4bdebfc8d07980c3d51f71ca44f1554ceeaf16877b3f0825f595dd"} Oct 29 22:15:32.891542 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.891512 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qq6r4" event={"ID":"1d3f4dde-95c6-41df-951b-316cd9d10f0a","Type":"ContainerStarted","Data":"680af67bdbfa9cdff8d076092e42778e5527522ad7b901ea4701739b293f52f2"} Oct 29 22:15:32.892512 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:32.892492 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmlxs" event={"ID":"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0","Type":"ContainerStarted","Data":"d0f311b79812b4cda2233f92166438c744447afdc5ad8d81b2d6a58cef8f556c"} Oct 29 22:15:33.470997 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:33.470957 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:33.614556 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:33.614522 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:33.899148 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:33.899108 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmlxs" event={"ID":"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0","Type":"ContainerStarted","Data":"80ec2a90c8c47e4a68eff315e894e9b5156319ec79ee4a5403469e1e8d684f38"} Oct 29 22:15:33.899331 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:33.899175 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmlxs" event={"ID":"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0","Type":"ContainerStarted","Data":"b36cf8ae19c20d6aee50657ae7c247f6e3c13414923e53b335dc8070dbae0521"} Oct 29 22:15:33.940893 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:33.940831 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nmlxs" podStartSLOduration=64.978056148 podStartE2EDuration="1m5.940817491s" totalImagesPullingTime="962.761343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:33.939673481 +0000 UTC m=+67.014990789" watchObservedRunningTime="2026-10-29 22:15:33.940817491 +0000 UTC m=+67.016134798" Oct 29 22:15:34.221449 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:34.221413 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:34.385092 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:34.385050 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:35.911372 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:35.911338 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qq6r4" event={"ID":"1d3f4dde-95c6-41df-951b-316cd9d10f0a","Type":"ContainerStarted","Data":"7f6ca3de66dcb28feb226a9c56c3c5be885c7b93427caf705c7a47d313e8dd82"} Oct 29 22:15:35.911811 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:35.911540 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:35.951130 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:35.950084 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qq6r4" podStartSLOduration=64.918666669 podStartE2EDuration="1m7.950067261s" totalImagesPullingTime="3.031400592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:35.94958029 +0000 UTC m=+69.024897599" watchObservedRunningTime="2026-10-29 22:15:35.950067261 +0000 UTC m=+69.025384571" Oct 29 22:15:36.836804 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:36.836763 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:36.972800 ip-10-0-133-251 kubenswrapper[2585]: I1029 22:15:36.972768 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:37.217623 ip-10-0-133-251 kubenswrapper[2585]: E1029 22:15:37.217595 2585 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:15:37.466856 ip-10-0-133-251 systemd[1]: Stopping Kubernetes Kubelet... Oct 29 22:15:37.477842 ip-10-0-133-251 systemd[1]: kubelet.service: Deactivated successfully. Oct 29 22:15:37.478008 ip-10-0-133-251 systemd[1]: Stopped Kubernetes Kubelet. Oct 29 22:15:37.478184 ip-10-0-133-251 systemd[1]: kubelet.service: Consumed 4.247s CPU time. Oct 29 22:15:37.487864 ip-10-0-133-251 systemd[1]: Starting Kubernetes Kubelet... Oct 29 22:15:37.549470 ip-10-0-133-251 kubenswrapper[5610]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:15:37.549470 ip-10-0-133-251 kubenswrapper[5610]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 29 22:15:37.549470 ip-10-0-133-251 kubenswrapper[5610]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:15:37.549470 ip-10-0-133-251 kubenswrapper[5610]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 29 22:15:37.551589 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551508 5610 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:37.551589 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551589 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:37.551589 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551593 5610 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551597 5610 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551600 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551603 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551612 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551615 5610 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551618 5610 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551621 5610 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551623 5610 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551626 5610 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551629 5610 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551632 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551634 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551637 5610 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551640 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551642 5610 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551645 5610 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551648 5610 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551650 5610 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551653 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:37.551685 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551655 5610 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551658 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551661 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551663 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551666 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551668 5610 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551671 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551674 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551677 5610 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551680 5610 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551684 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551687 5610 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551689 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551692 5610 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551695 5610 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551698 5610 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551700 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551704 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551707 5610 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551710 5610 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:37.552178 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551713 5610 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551716 5610 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551719 5610 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551721 5610 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551723 5610 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551726 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551728 5610 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551731 5610 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551734 5610 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551737 5610 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551740 5610 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551742 5610 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551745 5610 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551748 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551750 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551753 5610 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551755 5610 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551758 5610 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551760 5610 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551763 5610 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:37.552680 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551766 5610 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551769 5610 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551772 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551776 5610 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551781 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551785 5610 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551789 5610 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551792 5610 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551795 5610 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551798 5610 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551801 5610 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551804 5610 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551807 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551809 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551812 5610 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551815 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551817 5610 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551820 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:37.553164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551823 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551825 5610 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551828 5610 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551831 5610 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551834 5610 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551836 5610 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551839 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551842 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551844 5610 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551847 5610 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551849 5610 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551852 5610 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551855 5610 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551858 5610 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551861 5610 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551863 5610 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551869 5610 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551871 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551874 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551876 5610 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:37.553635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551879 5610 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551882 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551884 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551887 5610 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551889 5610 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.551898 5610 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552757 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552764 5610 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552767 5610 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552770 5610 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552773 5610 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552776 5610 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552780 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552782 5610 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552785 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552788 5610 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552790 5610 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552793 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552795 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552798 5610 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:37.554203 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552801 5610 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552803 5610 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552806 5610 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552809 5610 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552811 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552814 5610 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552816 5610 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552819 5610 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552824 5610 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552827 5610 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552830 5610 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552832 5610 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552835 5610 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552838 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552840 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552843 5610 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552846 5610 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552848 5610 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552851 5610 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552855 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:37.555577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552858 5610 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552861 5610 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552864 5610 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552866 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552869 5610 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552871 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552874 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552877 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552879 5610 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552882 5610 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552884 5610 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552887 5610 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552890 5610 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552892 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552895 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552898 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552900 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552903 5610 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552905 5610 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552908 5610 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:37.556073 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552912 5610 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552914 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552917 5610 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552919 5610 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552922 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552925 5610 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552927 5610 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552930 5610 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552932 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552935 5610 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552937 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552941 5610 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552943 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552946 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552948 5610 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552951 5610 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552953 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552956 5610 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552958 5610 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:37.556651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552961 5610 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552963 5610 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552966 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552968 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552971 5610 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552973 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552978 5610 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552981 5610 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552985 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552990 5610 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552994 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.552997 5610 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553000 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553004 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553007 5610 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553010 5610 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553012 5610 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553015 5610 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:37.557122 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553018 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553020 5610 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553023 5610 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553025 5610 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553029 5610 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553032 5610 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553034 5610 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553037 5610 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553040 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553042 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553045 5610 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553048 5610 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553050 5610 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553053 5610 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553056 5610 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553588 5610 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553595 5610 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553598 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553601 5610 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553604 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:37.557625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553606 5610 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553609 5610 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553612 5610 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553614 5610 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553617 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553620 5610 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553622 5610 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553626 5610 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553629 5610 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553632 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553634 5610 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553637 5610 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553640 5610 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553643 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553645 5610 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553648 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553651 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553654 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553657 5610 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:37.558160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553660 5610 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553663 5610 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553666 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553669 5610 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553671 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553674 5610 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553677 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553680 5610 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553682 5610 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553685 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553687 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553690 5610 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553692 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553695 5610 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553697 5610 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553699 5610 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553703 5610 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553707 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553710 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:37.558874 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553712 5610 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553715 5610 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553718 5610 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553720 5610 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553723 5610 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553725 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553728 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553730 5610 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553733 5610 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553736 5610 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553739 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553741 5610 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553744 5610 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553746 5610 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553750 5610 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553753 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553755 5610 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553758 5610 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553761 5610 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:37.559736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553763 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553766 5610 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553769 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553771 5610 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553774 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553776 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553779 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553782 5610 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553784 5610 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553787 5610 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553789 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553792 5610 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553794 5610 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553797 5610 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553799 5610 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553802 5610 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553804 5610 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553807 5610 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553809 5610 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553811 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:37.560660 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553814 5610 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553817 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553819 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553821 5610 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553824 5610 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553826 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553829 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553831 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553834 5610 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553837 5610 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553839 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553842 5610 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553845 5610 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553847 5610 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553850 5610 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553852 5610 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553855 5610 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553859 5610 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553863 5610 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553867 5610 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:37.561549 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553870 5610 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553873 5610 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553875 5610 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.553878 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554609 5610 flags.go:64] FLAG: --address="0.0.0.0" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554631 5610 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554639 5610 flags.go:64] FLAG: --anonymous-auth="true" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554645 5610 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554650 5610 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554653 5610 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554663 5610 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554668 5610 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554672 5610 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554676 5610 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554680 5610 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554683 5610 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554687 5610 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554690 5610 flags.go:64] FLAG: --cgroup-root="" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554696 5610 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554700 5610 flags.go:64] FLAG: --client-ca-file="" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554703 5610 flags.go:64] FLAG: --cloud-provider="external" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554706 5610 flags.go:64] FLAG: --cluster-dns="[]" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554711 5610 flags.go:64] FLAG: --cluster-domain="" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554715 5610 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 29 22:15:37.562448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554720 5610 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554724 5610 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554729 5610 flags.go:64] FLAG: --container-log-max-files="5" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554736 5610 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554739 5610 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554743 5610 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554746 5610 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554749 5610 flags.go:64] FLAG: --contention-profiling="false" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554753 5610 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554756 5610 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554760 5610 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554763 5610 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554770 5610 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554775 5610 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554778 5610 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554782 5610 flags.go:64] FLAG: --enable-load-reader="false" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554785 5610 flags.go:64] FLAG: --enable-server="true" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554789 5610 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554794 5610 flags.go:64] FLAG: --event-burst="100" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554798 5610 flags.go:64] FLAG: --event-qps="50" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554804 5610 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554808 5610 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554811 5610 flags.go:64] FLAG: --eviction-hard="" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554815 5610 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 29 22:15:37.563501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554819 5610 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554822 5610 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554825 5610 flags.go:64] FLAG: --eviction-soft="" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554829 5610 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554835 5610 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554838 5610 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554842 5610 flags.go:64] FLAG: --experimental-mounter-path="" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554845 5610 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554848 5610 flags.go:64] FLAG: --fail-swap-on="true" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554851 5610 flags.go:64] FLAG: --feature-gates="" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554855 5610 flags.go:64] FLAG: --file-check-frequency="20s" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554859 5610 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554862 5610 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554869 5610 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554876 5610 flags.go:64] FLAG: --healthz-port="10248" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554893 5610 flags.go:64] FLAG: --help="false" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554930 5610 flags.go:64] FLAG: --hostname-override="ip-10-0-133-251.ec2.internal" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.554958 5610 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555074 5610 flags.go:64] FLAG: --http-check-frequency="20s" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555081 5610 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555086 5610 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555090 5610 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555094 5610 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 29 22:15:37.564563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555097 5610 flags.go:64] FLAG: --image-service-endpoint="" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555100 5610 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555103 5610 flags.go:64] FLAG: --kube-api-burst="100" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555107 5610 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555110 5610 flags.go:64] FLAG: --kube-api-qps="50" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555113 5610 flags.go:64] FLAG: --kube-reserved="" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555117 5610 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555120 5610 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555123 5610 flags.go:64] FLAG: --kubelet-cgroups="" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555126 5610 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555130 5610 flags.go:64] FLAG: --lock-file="" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555133 5610 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555136 5610 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555139 5610 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555146 5610 flags.go:64] FLAG: --log-json-split-stream="false" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555150 5610 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555153 5610 flags.go:64] FLAG: --log-text-split-stream="false" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555156 5610 flags.go:64] FLAG: --logging-format="text" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555159 5610 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555163 5610 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555166 5610 flags.go:64] FLAG: --manifest-url="" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555169 5610 flags.go:64] FLAG: --manifest-url-header="" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555174 5610 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555177 5610 flags.go:64] FLAG: --max-open-files="1000000" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555182 5610 flags.go:64] FLAG: --max-pods="110" Oct 29 22:15:37.565593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555185 5610 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555188 5610 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555192 5610 flags.go:64] FLAG: --memory-manager-policy="None" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555196 5610 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555199 5610 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555202 5610 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555205 5610 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555213 5610 flags.go:64] FLAG: --node-status-max-images="50" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555216 5610 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555220 5610 flags.go:64] FLAG: --oom-score-adj="-999" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555223 5610 flags.go:64] FLAG: --pod-cidr="" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555226 5610 flags.go:64] FLAG: --pod-manifest-path="" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555229 5610 flags.go:64] FLAG: --pod-max-pids="-1" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555232 5610 flags.go:64] FLAG: --pods-per-core="0" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555235 5610 flags.go:64] FLAG: --port="10250" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555238 5610 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555241 5610 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0248937e7c6f384c7" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555245 5610 flags.go:64] FLAG: --qos-reserved="" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555248 5610 flags.go:64] FLAG: --read-only-port="10255" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555251 5610 flags.go:64] FLAG: --register-node="true" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555254 5610 flags.go:64] FLAG: --register-with-taints="" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555258 5610 flags.go:64] FLAG: --registry-burst="10" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555261 5610 flags.go:64] FLAG: --registry-qps="5" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555264 5610 flags.go:64] FLAG: --reserved-cpus="" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555270 5610 flags.go:64] FLAG: --reserved-memory="" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555278 5610 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 29 22:15:37.566703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555282 5610 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555285 5610 flags.go:64] FLAG: --rotate-certificates="false" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555288 5610 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555291 5610 flags.go:64] FLAG: --runonce="false" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555294 5610 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555297 5610 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555300 5610 flags.go:64] FLAG: --seccomp-default="false" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555303 5610 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555307 5610 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555310 5610 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555313 5610 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555316 5610 flags.go:64] FLAG: --storage-driver-password="root" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555319 5610 flags.go:64] FLAG: --storage-driver-secure="false" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555322 5610 flags.go:64] FLAG: --storage-driver-table="stats" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555325 5610 flags.go:64] FLAG: --storage-driver-user="root" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555328 5610 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555331 5610 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555334 5610 flags.go:64] FLAG: --system-cgroups="" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555337 5610 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555344 5610 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555347 5610 flags.go:64] FLAG: --tls-cert-file="" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555350 5610 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555355 5610 flags.go:64] FLAG: --tls-min-version="" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555360 5610 flags.go:64] FLAG: --tls-private-key-file="" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555363 5610 flags.go:64] FLAG: --topology-manager-policy="none" Oct 29 22:15:37.567707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555366 5610 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555369 5610 flags.go:64] FLAG: --topology-manager-scope="container" Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555372 5610 flags.go:64] FLAG: --v="2" Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555377 5610 flags.go:64] FLAG: --version="false" Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555381 5610 flags.go:64] FLAG: --vmodule="" Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555386 5610 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555428 5610 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555569 5610 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555573 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555576 5610 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555579 5610 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555582 5610 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555585 5610 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555587 5610 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555590 5610 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555594 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555597 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555601 5610 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555605 5610 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555609 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555612 5610 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:37.568437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555615 5610 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555618 5610 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555621 5610 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555624 5610 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555626 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555629 5610 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555632 5610 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555634 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555637 5610 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555642 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555645 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555648 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555651 5610 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555653 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555656 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555658 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555662 5610 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555665 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555669 5610 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555672 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:37.568978 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555675 5610 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555677 5610 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555680 5610 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555682 5610 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555685 5610 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555688 5610 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555690 5610 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555693 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555696 5610 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555698 5610 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555701 5610 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555704 5610 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555706 5610 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555709 5610 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555712 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555715 5610 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555717 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555720 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555722 5610 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555725 5610 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:37.569521 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555728 5610 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555731 5610 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555734 5610 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555737 5610 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555739 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555742 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555745 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555747 5610 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555751 5610 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555753 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555756 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555759 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555762 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555764 5610 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555767 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555769 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555772 5610 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555775 5610 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555777 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:37.570006 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555780 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555783 5610 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555785 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555788 5610 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555790 5610 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555793 5610 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555796 5610 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555798 5610 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555801 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555804 5610 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555806 5610 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555809 5610 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555811 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555814 5610 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555818 5610 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555820 5610 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555823 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555825 5610 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555828 5610 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555830 5610 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:37.570484 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555833 5610 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555837 5610 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555839 5610 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555842 5610 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555845 5610 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555848 5610 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555851 5610 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555853 5610 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555856 5610 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555859 5610 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555862 5610 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555866 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555869 5610 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.555896 5610 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555932 5610 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555937 5610 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:37.570971 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555940 5610 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555943 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555946 5610 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555949 5610 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555951 5610 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555954 5610 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555956 5610 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555960 5610 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555963 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555965 5610 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555970 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555973 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555975 5610 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555978 5610 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555980 5610 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555983 5610 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555985 5610 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555988 5610 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555991 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555994 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:37.571354 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.555997 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556000 5610 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556003 5610 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556005 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556008 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556011 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556014 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556016 5610 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556019 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556021 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556024 5610 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556026 5610 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556029 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556032 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556034 5610 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556037 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556040 5610 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556043 5610 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556045 5610 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:37.571969 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556048 5610 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556050 5610 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556053 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556057 5610 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556059 5610 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556062 5610 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556064 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556067 5610 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556070 5610 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556072 5610 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556075 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556079 5610 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556082 5610 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556084 5610 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556087 5610 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556090 5610 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556094 5610 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556096 5610 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556099 5610 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:37.572455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556101 5610 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556104 5610 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556107 5610 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556109 5610 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556112 5610 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556115 5610 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556118 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556121 5610 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556124 5610 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556127 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556129 5610 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556132 5610 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556134 5610 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556137 5610 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556139 5610 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556142 5610 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556146 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556149 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556153 5610 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:37.572934 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556157 5610 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556160 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556162 5610 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556165 5610 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556168 5610 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556171 5610 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556174 5610 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556176 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556179 5610 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556182 5610 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556185 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556188 5610 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556190 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556193 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556195 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556198 5610 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556201 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556203 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556207 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556210 5610 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:37.573408 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556212 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556215 5610 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556217 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556220 5610 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556223 5610 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556225 5610 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.556228 5610 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.560848 5610 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.560870 5610 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.560894 5610 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.560902 5610 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.560992 5610 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561002 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561009 5610 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561015 5610 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561020 5610 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561025 5610 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561029 5610 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561034 5610 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:37.574014 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561038 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561043 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561047 5610 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561051 5610 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561056 5610 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561061 5610 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561065 5610 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561069 5610 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561074 5610 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561078 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561082 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561087 5610 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561091 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561095 5610 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561100 5610 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561104 5610 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561109 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561113 5610 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561118 5610 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561122 5610 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:37.574781 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561127 5610 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561133 5610 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561137 5610 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561141 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561146 5610 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561150 5610 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561154 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561158 5610 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561162 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561167 5610 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561171 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561176 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561180 5610 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561184 5610 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561188 5610 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561193 5610 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561197 5610 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561201 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561205 5610 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561210 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:37.575539 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561214 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561218 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561223 5610 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561227 5610 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561231 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561235 5610 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561239 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561244 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561248 5610 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561253 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561260 5610 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561266 5610 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561273 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561278 5610 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561284 5610 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561288 5610 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561293 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561297 5610 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561301 5610 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:37.576027 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561305 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561310 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561314 5610 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561318 5610 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561323 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561327 5610 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561331 5610 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561335 5610 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561340 5610 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561345 5610 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561349 5610 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561353 5610 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561357 5610 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561361 5610 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561366 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561370 5610 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561374 5610 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561378 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561382 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:37.576634 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561387 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561407 5610 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561412 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561416 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561420 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561424 5610 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561429 5610 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561433 5610 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561437 5610 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561443 5610 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561447 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561451 5610 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561456 5610 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561460 5610 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561464 5610 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561469 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561473 5610 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561477 5610 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561481 5610 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:37.577268 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561486 5610 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.561528 5610 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561777 5610 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561786 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561792 5610 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561796 5610 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561802 5610 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561808 5610 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561813 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561817 5610 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561823 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561829 5610 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561836 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561841 5610 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561846 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:37.577779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561850 5610 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561855 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561859 5610 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561863 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561867 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561872 5610 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561876 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561880 5610 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561886 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561891 5610 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561895 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561899 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561903 5610 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561908 5610 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561912 5610 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561916 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561920 5610 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561928 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561933 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:37.578144 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561937 5610 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561941 5610 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561945 5610 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561950 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561954 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561958 5610 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561963 5610 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561967 5610 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561971 5610 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561976 5610 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561981 5610 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561985 5610 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561989 5610 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561993 5610 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.561998 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562002 5610 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562007 5610 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562011 5610 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562016 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562020 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:37.578635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562024 5610 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562029 5610 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562034 5610 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562038 5610 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562042 5610 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562047 5610 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562051 5610 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562055 5610 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562060 5610 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562064 5610 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562070 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562075 5610 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562080 5610 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562084 5610 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562088 5610 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562093 5610 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562097 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562102 5610 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562106 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:37.579136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562110 5610 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562115 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562119 5610 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562123 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562127 5610 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562131 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562135 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562140 5610 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562144 5610 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562148 5610 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562154 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562159 5610 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562164 5610 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562168 5610 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562172 5610 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562176 5610 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562182 5610 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562187 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562191 5610 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562195 5610 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:37.579636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562200 5610 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562205 5610 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562209 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562215 5610 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562220 5610 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562224 5610 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562228 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562233 5610 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562237 5610 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562241 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562246 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562250 5610 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562254 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562258 5610 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562262 5610 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.562299 5610 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 29 22:15:37.580136 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562351 5610 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562357 5610 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562362 5610 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562366 5610 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562371 5610 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562376 5610 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562384 5610 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562405 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562411 5610 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562415 5610 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562419 5610 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562424 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562428 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562434 5610 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562438 5610 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562442 5610 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562446 5610 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562450 5610 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562455 5610 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 29 22:15:37.580645 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562458 5610 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562463 5610 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562467 5610 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562471 5610 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562476 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562480 5610 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562484 5610 feature_gate.go:386] unrecognized feature gate: Example2 Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562488 5610 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562492 5610 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562497 5610 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562501 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562506 5610 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562510 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562514 5610 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562518 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562522 5610 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562526 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562530 5610 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562534 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 29 22:15:37.581159 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562539 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562543 5610 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562547 5610 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562551 5610 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562556 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562560 5610 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562564 5610 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562569 5610 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562574 5610 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562578 5610 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562583 5610 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562587 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562591 5610 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562595 5610 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562599 5610 feature_gate.go:386] unrecognized feature gate: Example Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562603 5610 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562608 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562611 5610 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562616 5610 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562621 5610 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 29 22:15:37.581654 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562626 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562631 5610 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562635 5610 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562639 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562644 5610 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562648 5610 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562652 5610 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562656 5610 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562660 5610 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562664 5610 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562669 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562673 5610 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562677 5610 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562684 5610 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562690 5610 feature_gate.go:386] unrecognized feature gate: EVPN Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562694 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562698 5610 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562702 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562709 5610 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 29 22:15:37.582158 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562715 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562719 5610 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562724 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562729 5610 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562734 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562738 5610 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562742 5610 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562746 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562751 5610 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562755 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562759 5610 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562763 5610 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562769 5610 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562774 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562778 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562782 5610 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562786 5610 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562790 5610 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562795 5610 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 29 22:15:37.582663 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562799 5610 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562804 5610 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562808 5610 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562812 5610 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562816 5610 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562820 5610 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562825 5610 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562829 5610 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562833 5610 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:15:37.562837 5610 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.562930 5610 server.go:957] "Client rotation is on, will bootstrap in background" Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.564455 5610 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.564528 5610 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.564777 5610 server.go:1014] "Starting client certificate rotation" Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.564911 5610 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.564959 5610 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:09:28 +0000 UTC" deadline="2028-06-03 04:02:23.581825988 +0000 UTC" Oct 29 22:15:37.583213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.564985 5610 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13973h46m46.01684244s" Oct 29 22:15:37.583673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.565509 5610 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:15:37.583673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.567145 5610 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 29 22:15:37.583673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.568662 5610 log.go:25] "Validated CRI v1 runtime API" Oct 29 22:15:37.583673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.570471 5610 log.go:25] "Validated CRI v1 image API" Oct 29 22:15:37.583673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.571149 5610 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 29 22:15:37.583673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.573176 5610 fs.go:135] Filesystem UUIDs: map[5a12698b-df65-4e0a-8d1f-b21dc75069d0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ec15bfe1-5bd9-4324-b832-4a4297747a3c:/dev/nvme0n1p4] Oct 29 22:15:37.583895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.573196 5610 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/105e933c13d5663a26543f97ec22b92ca3bb2edef90ab4abea7c5d64fe1eadf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/105e933c13d5663a26543f97ec22b92ca3bb2edef90ab4abea7c5d64fe1eadf2/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22b3d2a8e71ef4dfe0ffae07c71ca79169936720dd8fd257539805f3c409cea6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22b3d2a8e71ef4dfe0ffae07c71ca79169936720dd8fd257539805f3c409cea6/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26b299caf779a2a40922c91faf822823b3c1d619edb455491ac93f506c234cf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26b299caf779a2a40922c91faf822823b3c1d619edb455491ac93f506c234cf0/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/495cd647124ee7f1bb17ba294b19464e4ae192eeafa4b1623c98792cb046bcc4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/495cd647124ee7f1bb17ba294b19464e4ae192eeafa4b1623c98792cb046bcc4/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/506e60b79a4bdebfc8d07980c3d51f71ca44f1554ceeaf16877b3f0825f595dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/506e60b79a4bdebfc8d07980c3d51f71ca44f1554ceeaf16877b3f0825f595dd/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52c3a9dc13a1dba0fd34874033ba5a08d028a086027458a96c674fae39bd1ea3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52c3a9dc13a1dba0fd34874033ba5a08d028a086027458a96c674fae39bd1ea3/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54828e6587593c8fe349bb25a9f11d49ea0da33a7e857275255c5ec5170c972a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54828e6587593c8fe349bb25a9f11d49ea0da33a7e857275255c5ec5170c972a/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/558717eb74111b51c72a34af66623138e8f8d2d6cf6bbee0dddfe1bd309de5d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/558717eb74111b51c72a34af66623138e8f8d2d6cf6bbee0dddfe1bd309de5d4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/680af67bdbfa9cdff8d076092e42778e5527522ad7b901ea4701739b293f52f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/680af67bdbfa9cdff8d076092e42778e5527522ad7b901ea4701739b293f52f2/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d845444a56b8edc586e7039acc0ab3acfa7ff6c2fb1b73ea5754089bfc0f63c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d845444a56b8edc586e7039acc0ab3acfa7ff6c2fb1b73ea5754089bfc0f63c/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/705c2f09cd0cbcdf4d8830dfaa7b519fdea6cebe904bc25737de615723ad0ed8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/705c2f09cd0cbcdf4d8830dfaa7b519fdea6cebe904bc25737de615723ad0ed8/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7931779220d01a3c9e6706d456b9431572ac5cb47009367ae9fb0f5c43541e8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7931779220d01a3c9e6706d456b9431572ac5cb47009367ae9fb0f5c43541e8a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7eee30eb857c093ae68fcc631f38870418c1ea438d4d56ea3e50225dac2c09dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7eee30eb857c093ae68fcc631f38870418c1ea438d4d56ea3e50225dac2c09dd/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8432b647b5c4d3a0af1525c48fc1e0420a6dcc38532b8827781a9c9be3ec3b03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8432b647b5c4d3a0af1525c48fc1e0420a6dcc38532b8827781a9c9be3ec3b03/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abaf932855fdfcd88bf7d4c28ff3df7833181ea8cbe046a483d1c7a6e48869be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abaf932855fdfcd88bf7d4c28ff3df7833181ea8cbe046a483d1c7a6e48869be/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2c1fb0200c80168207b2c84d2d237d12c86dc4d8773b20cd5cfa5566552055e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2c1fb0200c80168207b2c84d2d237d12c86dc4d8773b20cd5cfa5566552055e/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca593fa5fc933d17b00aa85fa018eefe914c1acd4923f704d4174a1635ef70af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca593fa5fc933d17b00aa85fa018eefe914c1acd4923f704d4174a1635ef70af/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0f311b79812b4cda2233f92166438c744447afdc5ad8d81b2d6a58cef8f556c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0f311b79812b4cda2233f92166438c744447afdc5ad8d81b2d6a58cef8f556c/userdata/shm major:0 minor:416 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d428635318497c8cd054862e7ff80171b2e5accdc73366c679a8bfa7f91dd5de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d428635318497c8cd054862e7ff80171b2e5accdc73366c679a8bfa7f91dd5de/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6b9149c5d7276214044a17e1af22c9e6935502e58ad3e74c46ae8b1cec067a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6b9149c5d7276214044a17e1af22c9e6935502e58ad3e74c46ae8b1cec067a5/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa41961e658bac9ac8d71a1ff36164e523e435257775628b5b2bee14e1ed55fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa41961e658bac9ac8d71a1ff36164e523e435257775628b5b2bee14e1ed55fc/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fca0ff5a75c41a1042824b1c9cacd3386cfada13220a4c74a25ad1f0a19d1764/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fca0ff5a75c41a1042824b1c9cacd3386cfada13220a4c74a25ad1f0a19d1764/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/config-reloader/2 major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/prometheus/15 major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~empty-dir/config-out major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~projected/kube-api-access-27m6m:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~projected/kube-api-access-27m6m major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~projected/tls-assets major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/config major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/web-config major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f585004-5478-4841-af5c-81c1c4bd9072/volumes/kubernetes.io~projected/kube-api-access-jnncl:{mountpoint:/var/lib/kubelet/pods/0f585004-5478-4841-af5c-81c1c4bd9072/volumes/kubernetes.io~projected/kube-api-access-jnncl major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f585004-5478-4841-af5c-81c1c4bd9072/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0f585004-5478-4841-af5c-81c1c4bd9072/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d3f4dde-95c6-41df-951b-316cd9d10f0a/volumes/kubernetes.io~projected/kube-api-access-zn6ww:{mountpoint:/var/lib/kubelet/pods/1d3f4dde-95c6-41df-951b-316cd9d10f0a/volumes/kubernetes.io~projected/kube-api-access-zn6ww major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35dc91f0-4639-4294-a296-418c22a692c1/volumes/kubernetes.io~projected/kube-api-access-d8769:{mountpoint:/var/lib/kubelet/pods/35dc91f0-4639-4294-a296-418c22a692c1/volumes/kubernetes.io~projected/kube-api-access-d8769 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b731a32-138f-4eb8-830a-7078d3667e68/volumes/kubernetes.io~projected/kube-api-access-6dtsh:{mountpoint:/var/lib/kubelet/pods/3b731a32-138f-4eb8-830a-7078d3667e68/volumes/kubernetes.io~projected/kube-api-access-6dtsh major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b731a32-138f-4eb8-830a-7078d3667e68/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3b731a32-138f-4eb8-830a-7078d3667e68/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09/volumes/kubernetes.io~projected/kube-api-access-sd8md:{mountpoint:/var/lib/kubelet/pods/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09/volumes/kubernetes.io~projected/kube-api-access-sd8md major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~projected/kube-api-access-mwqh2:{mountpoint:/var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~projected/kube-api-access-mwqh2 major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~projected/kube-api-access-pj8m5:{mountpoint:/var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~projected/kube-api-access-pj8m5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volumes/kubernetes.io~projected/kube-api-access-9zrr9:{mountpoint:/var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volumes/kubernetes.io~projected/kube-api-access-9zrr9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~projected/kube-api-access-tfcc9:{mountpoint:/var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~projected/kube-api-access-tfcc9 major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~projected/kube-api-access-g6r2d:{mountpoint:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~projected/kube-api-access-g6r2d major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~projected/kube-api-access-cb77h:{mountpoint:/var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~projected/kube-api-access-cb77h major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a06ed94e-63bb-4a18-b959-4175a0972ffb/volumes/kubernetes.io~projected/kube-api-access-k88ts:{mountpoint:/var/lib/kubelet/pods/a06ed94e-63bb-4a18-b959-4175a0972ffb/volumes/kubernetes.io~projected/kube-api-access-k88ts major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a06ed94e-63bb-4a18-b959-4175a0972ffb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a06ed94e-63bb-4a18-b959-4175a0972ffb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa39a765-7c7f-4212-a8e2-c7f2a91f7339/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/aa39a765-7c7f-4212-a8e2-c7f2a91f7339/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab618391-4607-4f00-ab02-819dc9897da1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ab618391-4607-4f00-ab02-819dc9897da1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3b72e10-5245-4d8e-9699-bae1c061e336/volumes/kubernetes.io~projected/kube-api-access-7zddq:{mountpoint:/var/lib/kubelet/pods/b3b72e10-5245-4d8e-9699-bae1c061e336/volumes/kubernetes.io~projected/kube-api-access-7zddq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0/volumes/kubernetes.io~projected/kube-api-access-shpll:{mountpoint:/var/lib/kubelet/pods/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0/volumes/kubernetes.io~projected/kube-api-access-shpll major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0/volumes/kubernetes.io~secret/metrics-certs major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc7d661f-6060-41b9-a442-ab840f6129e5/volumes/kubernetes.io~projected/kube-api-access-2wjvs:{mountpoint:/var/lib/kubelet/pods/cc7d661f-6060-41b9-a442-ab840f6129e5/volumes/kubernetes.io~projected/kube-api-access-2wjvs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af/volumes/kubernetes.io~projected/kube-api-access-wn28n:{mountpoint:/var/lib/kubelet/pods/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af/volumes/kubernetes.io~projected/kube-api-access-wn28n major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79/volumes/kubernetes.io~projected/kube-api-access-n425l:{mountpoint:/var/lib/kubelet/pods/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79/volumes/kubernetes.io~projected/kube-api-access-n425l major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1daa8f6-7854-4c2a-b207-bd6947f3e030/volumes/kubernetes.io~projected/kube-api-access-hrx4w:{mountpoint:/var/lib/kubelet/pods/e1daa8f6-7854-4c2a-b207-bd6947f3e030/volumes/kubernetes.io~projected/kube-api-access-hrx4w major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/475785b1ae2fbe9b09490f6e6dc250d4f167467d2fdd939fb3acf389932cddaf/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/06d9e358d8c23a2b3c04b00e535d355aefff50bd1331d68349ea491eb7bcc087/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5f364189bb01dc3e35f0fd1ac16cbe6fc47d4d9c8ac7a8b2a29fdc4a6ca18084/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/9dbfe62c86e07f89df6012bf14ebc6bd6ee1cc33567c8ed747c4beb09a876a25/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a9651f31e8f6250c036f7ab2414fa8ce1a4ef9384ad98741ee540b97f870fdf2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c14f3c675aa351f2fce4acb905b6fa7e5fe2bab9f16a6b2a057c99a89401f8af/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/213746c22133ad0f456fb3e204826f1f1977f4968874eeeeadc75a9e1e0cc06f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/00771a3fdac0f4588ef8dea358148280706cdd028bf3518dbcf5cab6943c807e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/6531124e9ce88efd1407b4705330b6f312912f8a463dbeaa30df945023eed4ed/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/3d380c91d238892795d88f87433a298a6933a1552cf8440aa662c3b3833d30d0/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e7961328b9ef0bbac75450960252438dafe2279f7e0317e5ce600dda568611ed/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/0054e50c207f4d1a02952ed689f62053a55045c02f859926370fa564e7d11b69/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7a1e4717a4ef4d21b700ffcb836f0e5ccacfa02e50032abe6a6319fc7658030f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/f487df96fddaca70b5b0a0ae7b16e0bd2cedffaee3d221ed2ab6c0a335a97010/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ad9a4acb9c77f5079c257fc8382e3816da2b88fb70476820c4822c121d49c7e4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/d2df54fa92f39a9691b5ccfd2d09dc2d0f8e924cf83797fef7d9db15af63c302/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ddadc8bdfc3fd58e458bdc5b01368de6691b33c82d3462a4e349a79a583b6508/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cb84de79dd66ea909521045206d0f3c13a372ed2bd6fbbd21588ef08f78ca981/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/02c4c6b50ba52e22052605326454242e35e55e79ad28152a2bc4850abd995e4f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fd26ca99879c6ae4cfeaa2ee8d60122ceb4c71e02d8ca035307818a7a5b12053/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/94b4603b983ec0750d80ae84280c7aabdc313435bc5fbef919a2e44182f1d92c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/8132370e0063a9f748e1522b3a22911d1bfb94a6248c01ab2645fb40ba3554bf/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f284e450fde243581be5e509cbebc915d408783743bd71d1a0d7d1837b8dde41/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/2ab5a776c73d2c8ac97b09315fed2ce834768593e6146591670c2a7fb90376e3/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/a6cb9cc6b4cec9ce3e78a2381bb3ee12f7fda0ee9bfdd2744861f6e0fe95c916/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/c9a3468e5fb108da14ecf071fbd98a78066b2af3fa4bd9c24fb85ca5990bb00a/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/558823485d348966c581f64193ac98aa1d3732849718e175b5cad381173302ca/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/87947b52cff6d9e78d03a9f601efb8e7cf55cca4cce8faf327d003e378f53c3f/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/ce2ebefdaae31647efdc92ffa718408939e5addfa7a1d75d11d5a6db726166eb/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/fcc6d5f9cfbab770d44944b7f3caf37d9035cfda72523c6d768a18bbcf4421bc/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/74210c16763cfac22c15be70c9ef955daa3109c13bdef9468846f8ea40e72566/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/27e930667eb0babefdd9bc53b28fe637390855ebb23b5cc4276d01c1dfac2a93/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/17599d5a4cdd63c9517c93e766c44694f763764b474b83dbe1fc7a0bc899bdfc/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/dc45e999db3f39f3846873429440c4e329594aa73aa8c5dfdc4fcc05a7121dff/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/5f1514588bed18eacc8e7f2b93d4206adbc00c227e43176ee4fc832e48988142/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/c5de6cde559f7db68f1e24c39b4933dbcd274f4b835fc166148ec18de7566fa5/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/45af179e0565f14b93e2b896867306d4ffee7134e465d3963514ac2ee6435a42/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/312e441882cc36b4b0509ee4618c1dcbe9daac8d9e0be614d73c0243118366a5/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/729152e5bd7abf1de0042c35fd66cbcd5baa2fcc700e02a4b13e2e6b2e80b837/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/0f4d641fc9a5a30f010081fc62a736eb6aa90684ce97332253ebf5b62bb9df38/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/84f88056d82f78c5db4a48dc49724289236fae762041f43a30ec3f46c1eb8485/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/307b083c0ddf84165c50f70ae1225fabc60d00a0c9c285a8bd10189fdc136827/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/debe359593a6de2da57231e0fb9b57b38467345f9ccf39c97898ff0733930bbc/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/dd70a0e851d70da3fba336819f20a637648fe95002463b785b502af676298108/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/c31f73ffe08b25aeee3fadc6f6abd1f09823817e06066895972dcec7090c2270/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/13a6d0ad44d1b79b673aca0c9f7f311b506f03b7a95f0fe5734a91013647f430/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/356ac48fc2278da1697447077b63e2e97f40066d4b77f1c4bd6146a5b7eb7233/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/617fdab8eae6046683dd83c4b4d5c044a19faec7026228262bddde34b420a17e/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/08c847701a32a4ad22acc24bd64b795fd3e29daab43cd858fad6f028fed0f105/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/e98cb78d73a170a1ae7471375fa9c36d9a5addfbf45b45f72d3a33763ae9488f/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/f55e4b0e03be7dd875fc277ee6656556034cf109d73492170c9ddc78391f5d11/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/11ea0cf82c0a7f00bc74767fe5253c00a2e16f67da60a2297ff639eeb8318e05/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/33cba3f49f08ac1c83e2b30e7ecf3ab4a5b67c36068d3af7687451a813c79b80/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/7600daaa354cb2167b412a349dffcdda4b675405a29a2a157896d14edf5b99bb/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/bc6d2656b3419b490d2929babe39be3703660966a5b78f37ba8160cf3f8ad7de/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/1c75a2f9f8c1a58ee2c433a6ae3b2b5835f0db6fa139c926ef734a526f654fa4/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/d8ef96a24262783535f2447b043200fb6e3eb967780557c5a7698e3db0091281/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/66e2174e8e63cc1f59a8334f10b2f48b6085d3e6bb96a6023122ce70a0bfdb60/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/6af9a0d87a5df0d671b42c56df5754158f42b7a43bdc717e6feef5e4d9c57eaf/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/edbd924876e6923ac84bacbd1051b1f4df5990c1ace9b8e58844d7a8edaf6543/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/cefc37f216b5730eeb4f4814b927b0da41cc33938e432a73919959cb0532093c/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/f564884e15f1ddb736e1e00e1505d7b098cca5dcaab527cadc887f0c1d013310/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/2469924f6f0d8a40301448e345e840fa6afc9373d03a2eae5af519ab3147dffb/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/8539ff6cc4ac4be90a997634590e2bec70153fc3188230c59279eadad989a378/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/d41b86a0a5173e8450946ac64320dbbde2c21aef624d38de9e2e1884d2b2871c/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1118b6fb04c8039d4768bd2a744c2277b31e8ddfa5cf71cdbecdbd89675bea04/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/e1625e11619d8a7e8dea344b3fe052173aa0af45d21a6a00c1b0e76096076532/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9c22541d30b43653945f154bfbc03b9a521a4845efc7c153b64c91bac3d161a0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/00f28076c9857fdffdd56f1b2fdb75eb9420fdafb93cfb940271b2a13d7d19c4/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5827e5456d073313f1763441697399f8d7524539ee13dbd9a927502138685381/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a13d088576b5634283bae0562c771a95a6e715c4ae9fb24db06553e08d0262b1/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2f9ddba478ae8d0fba68ee3ec8a25e6c8ff49fe80c653805bc5cff506c2b7a92/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/2c7a97816fed0bf270d4c0c3b47d35a26f546a9e4f67894c33df8eeeedba91d7/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/80bb129c8de2cdeba86d049de0348b1336c6934465c31764a9542d6d32a6c42b/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.581563 5610 manager.go:217] Machine: {Timestamp:2026-10-29 22:15:37.581080354 +0000 UTC m=+0.062397839 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116295 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c9602487d30bccfadc9bf11dfbebf SystemUUID:ec2c9602-487d-30bc-cfad-c9bf11dfbebf BootID:c5a5a7f6-a179-45c7-a75d-f608ba40aace Filesystems:[{Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:411 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3b72e10-5245-4d8e-9699-bae1c061e336/volumes/kubernetes.io~projected/kube-api-access-7zddq DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/35dc91f0-4639-4294-a296-418c22a692c1/volumes/kubernetes.io~projected/kube-api-access-d8769 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~projected/kube-api-access-g6r2d DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca593fa5fc933d17b00aa85fa018eefe914c1acd4923f704d4174a1635ef70af/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e1daa8f6-7854-4c2a-b207-bd6947f3e030/volumes/kubernetes.io~projected/kube-api-access-hrx4w DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~projected/kube-api-access-tfcc9 DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:366 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa41961e658bac9ac8d71a1ff36164e523e435257775628b5b2bee14e1ed55fc/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0f585004-5478-4841-af5c-81c1c4bd9072/volumes/kubernetes.io~projected/kube-api-access-jnncl DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b731a32-138f-4eb8-830a-7078d3667e68/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0f311b79812b4cda2233f92166438c744447afdc5ad8d81b2d6a58cef8f556c/userdata/shm DeviceMajor:0 DeviceMinor:416 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d428635318497c8cd054862e7ff80171b2e5accdc73366c679a8bfa7f91dd5de/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6b9149c5d7276214044a17e1af22c9e6935502e58ad3e74c46ae8b1cec067a5/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79/volumes/kubernetes.io~projected/kube-api-access-n425l DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ab618391-4607-4f00-ab02-819dc9897da1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:415 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~projected/kube-api-access-pj8m5 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d3f4dde-95c6-41df-951b-316cd9d10f0a/volumes/kubernetes.io~projected/kube-api-access-zn6ww DeviceMajor:0 DeviceMinor:418 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f585004-5478-4841-af5c-81c1c4bd9072/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~projected/kube-api-access-mwqh2 DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d845444a56b8edc586e7039acc0ab3acfa7ff6c2fb1b73ea5754089bfc0f63c/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abaf932855fdfcd88bf7d4c28ff3df7833181ea8cbe046a483d1c7a6e48869be/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/558717eb74111b51c72a34af66623138e8f8d2d6cf6bbee0dddfe1bd309de5d4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a06ed94e-63bb-4a18-b959-4175a0972ffb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/680af67bdbfa9cdff8d076092e42778e5527522ad7b901ea4701739b293f52f2/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7931779220d01a3c9e6706d456b9431572ac5cb47009367ae9fb0f5c43541e8a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22b3d2a8e71ef4dfe0ffae07c71ca79169936720dd8fd257539805f3c409cea6/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/105e933c13d5663a26543f97ec22b92ca3bb2edef90ab4abea7c5d64fe1eadf2/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09/volumes/kubernetes.io~projected/kube-api-access-sd8md DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/705c2f09cd0cbcdf4d8830dfaa7b519fdea6cebe904bc25737de615723ad0ed8/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2c1fb0200c80168207b2c84d2d237d12c86dc4d8773b20cd5cfa5566552055e/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~projected/kube-api-access-27m6m DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54828e6587593c8fe349bb25a9f11d49ea0da33a7e857275255c5ec5170c972a/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52c3a9dc13a1dba0fd34874033ba5a08d028a086027458a96c674fae39bd1ea3/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/629cfb60-8afd-46e7-86a8-1de336684f20/volumes/kubernetes.io~projected/kube-api-access-9zrr9 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bfd4dd9-a2a6-4ecd-9667-840738787716/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71813e11-80fa-4c04-ba56-cd1bc57d3f35/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0/volumes/kubernetes.io~projected/kube-api-access-shpll DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc7d661f-6060-41b9-a442-ab840f6129e5/volumes/kubernetes.io~projected/kube-api-access-2wjvs DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af/volumes/kubernetes.io~projected/kube-api-access-wn28n DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7eee30eb857c093ae68fcc631f38870418c1ea438d4d56ea3e50225dac2c09dd/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26b299caf779a2a40922c91faf822823b3c1d619edb455491ac93f506c234cf0/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:244 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c71f14d-b3bf-4942-beaa-6cdf18e331b9/volumes/kubernetes.io~projected/kube-api-access-cb77h DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/aa39a765-7c7f-4212-a8e2-c7f2a91f7339/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82b16e7a-7a16-409f-9411-d23156de1fe1/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8432b647b5c4d3a0af1525c48fc1e0420a6dcc38532b8827781a9c9be3ec3b03/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fca0ff5a75c41a1042824b1c9cacd3386cfada13220a4c74a25ad1f0a19d1764/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3f6e07a6-8957-44f5-9203-425259d84060/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/495cd647124ee7f1bb17ba294b19464e4ae192eeafa4b1623c98792cb046bcc4/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a06ed94e-63bb-4a18-b959-4175a0972ffb/volumes/kubernetes.io~projected/kube-api-access-k88ts DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3b731a32-138f-4eb8-830a-7078d3667e68/volumes/kubernetes.io~projected/kube-api-access-6dtsh DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:368 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/506e60b79a4bdebfc8d07980c3d51f71ca44f1554ceeaf16877b3f0825f595dd/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:26b299caf779a2a MacAddress:ba:60:95:cf:70:cb Speed:10000 Mtu:8901} {Name:495cd647124ee7f MacAddress:9e:6b:68:c9:37:d0 Speed:10000 Mtu:8901} {Name:506e60b79a4bdeb MacAddress:1a:98:a2:97:43:1b Speed:10000 Mtu:8901} {Name:52c3a9dc13a1dba MacAddress:0a:63:e3:a3:96:22 Speed:10000 Mtu:8901} {Name:680af67bdbfa9cd MacAddress:b2:e2:b4:0d:54:35 Speed:10000 Mtu:8901} {Name:6d845444a56b8ed MacAddress:a6:f8:8f:dd:e2:a8 Speed:10000 Mtu:8901} {Name:8432b647b5c4d3a MacAddress:4a:e4:12:f9:c9:3b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:a9:ac:ed Speed:0 Mtu:9001} {Name:br-int MacAddress:02:b1:4a:3c:95:02 Speed:0 Mtu:8901} {Name:c2c1fb0200c8016 MacAddress:6e:86:1a:60:0e:c7 Speed:10000 Mtu:8901} {Name:ca593fa5fc933d1 MacAddress:7a:ec:62:6d:01:46 Speed:10000 Mtu:8901} {Name:d0f311b79812b4c MacAddress:66:4a:6a:60:fc:31 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:a9:ac:ed Speed:0 Mtu:9001} {Name:f6b9149c5d72762 MacAddress:82:0e:09:8a:79:02 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6e:85:eb:8b:01:fb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:94:9e:07:7c:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.581975 5610 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582019 5610 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582251 5610 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582273 5610 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582441 5610 topology_manager.go:143] "Creating topology manager with none policy" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582449 5610 container_manager_linux.go:311] "Creating device plugin manager" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582461 5610 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582479 5610 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582570 5610 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582719 5610 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582819 5610 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582850 5610 kubelet.go:493] "Attempting to sync node with API server" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582863 5610 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582872 5610 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582881 5610 kubelet.go:405] "Adding apiserver pod source" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.582897 5610 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.583748 5610 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:15:37.584210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.583762 5610 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:15:37.584946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.584284 5610 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 29 22:15:37.587197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587182 5610 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 29 22:15:37.587197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587195 5610 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 29 22:15:37.587308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587298 5610 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 29 22:15:37.587516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587504 5610 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 29 22:15:37.587553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587534 5610 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 29 22:15:37.587685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587673 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 29 22:15:37.587714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587699 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 29 22:15:37.587714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587711 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 29 22:15:37.587765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587719 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 29 22:15:37.587765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587725 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 29 22:15:37.587765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587731 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 29 22:15:37.587765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587736 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 29 22:15:37.587765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587742 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 29 22:15:37.587765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587748 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 29 22:15:37.587765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587753 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 29 22:15:37.587765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587762 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 29 22:15:37.587990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587770 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 29 22:15:37.587990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587796 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 29 22:15:37.587990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.587804 5610 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 29 22:15:37.591293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.591280 5610 server.go:1263] "Started kubelet" Oct 29 22:15:37.591380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.591325 5610 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 29 22:15:37.591848 ip-10-0-133-251 systemd[1]: Started Kubernetes Kubelet. Oct 29 22:15:37.592243 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.591976 5610 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 29 22:15:37.592352 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.592338 5610 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 29 22:15:37.592908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.592792 5610 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.594544 5610 server.go:317] "Adding debug handlers to kubelet server" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.597659 5610 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.597686 5610 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.597745 5610 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:09:29 +0000 UTC" deadline="2028-05-26 16:45:17.994400366 +0000 UTC" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.597765 5610 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13794h29m40.396638847s" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.597769 5610 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.597782 5610 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:37.597798 5610 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found" Oct 29 22:15:37.598622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.597867 5610 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 29 22:15:37.599377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.599355 5610 factory.go:55] Registering systemd factory Oct 29 22:15:37.599377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.599377 5610 factory.go:223] Registration of the systemd container factory successfully Oct 29 22:15:37.599616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.599599 5610 factory.go:153] Registering CRI-O factory Oct 29 22:15:37.599616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.599618 5610 factory.go:223] Registration of the crio container factory successfully Oct 29 22:15:37.599773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.599676 5610 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 29 22:15:37.599773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.599696 5610 factory.go:103] Registering Raw factory Oct 29 22:15:37.599773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.599711 5610 manager.go:1199] Started watching for new ooms in manager Oct 29 22:15:37.600097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600084 5610 manager.go:319] Starting recovery of all containers Oct 29 22:15:37.600420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600357 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca" seLinuxMountContext="" Oct 29 22:15:37.600488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600430 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 29 22:15:37.600488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600453 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls" seLinuxMountContext="" Oct 29 22:15:37.600488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600469 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b731a32-138f-4eb8-830a-7078d3667e68" volumeName="kubernetes.io/secret/3b731a32-138f-4eb8-830a-7078d3667e68-cert" seLinuxMountContext="" Oct 29 22:15:37.600488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600483 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09" volumeName="kubernetes.io/projected/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-kube-api-access-sd8md" seLinuxMountContext="" Oct 29 22:15:37.600598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600495 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f6e07a6-8957-44f5-9203-425259d84060" volumeName="kubernetes.io/projected/3f6e07a6-8957-44f5-9203-425259d84060-kube-api-access-mwqh2" seLinuxMountContext="" Oct 29 22:15:37.600598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600507 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfd4dd9-a2a6-4ecd-9667-840738787716" volumeName="kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-tuned" seLinuxMountContext="" Oct 29 22:15:37.600598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600518 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfd4dd9-a2a6-4ecd-9667-840738787716" volumeName="kubernetes.io/projected/4bfd4dd9-a2a6-4ecd-9667-840738787716-kube-api-access-pj8m5" seLinuxMountContext="" Oct 29 22:15:37.600598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600531 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 29 22:15:37.600598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600542 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 29 22:15:37.600598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600554 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 29 22:15:37.600598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600567 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa39a765-7c7f-4212-a8e2-c7f2a91f7339" volumeName="kubernetes.io/configmap/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-konnectivity-ca" seLinuxMountContext="" Oct 29 22:15:37.600598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600579 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab618391-4607-4f00-ab02-819dc9897da1" volumeName="kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600606 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" volumeName="kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600620 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0c9a67d-21dc-44cf-ab1f-1ef20fba70af" volumeName="kubernetes.io/projected/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-kube-api-access-wn28n" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600645 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600659 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dc91f0-4639-4294-a296-418c22a692c1" volumeName="kubernetes.io/projected/35dc91f0-4639-4294-a296-418c22a692c1-kube-api-access-d8769" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600673 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b731a32-138f-4eb8-830a-7078d3667e68" volumeName="kubernetes.io/projected/3b731a32-138f-4eb8-830a-7078d3667e68-kube-api-access-6dtsh" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600686 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfd4dd9-a2a6-4ecd-9667-840738787716" volumeName="kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-tmp" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600700 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06ed94e-63bb-4a18-b959-4175a0972ffb" volumeName="kubernetes.io/empty-dir/a06ed94e-63bb-4a18-b959-4175a0972ffb-data-volume" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600713 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600727 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600739 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71813e11-80fa-4c04-ba56-cd1bc57d3f35" volumeName="kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-tls" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600754 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc7d661f-6060-41b9-a442-ab840f6129e5" volumeName="kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-cni-binary-copy" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600768 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0c9a67d-21dc-44cf-ab1f-1ef20fba70af" volumeName="kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600780 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="629cfb60-8afd-46e7-86a8-1de336684f20" volumeName="kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-config" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600793 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-grpc-tls" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600806 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c71f14d-b3bf-4942-beaa-6cdf18e331b9" volumeName="kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-tls" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600828 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc7d661f-6060-41b9-a442-ab840f6129e5" volumeName="kubernetes.io/projected/cc7d661f-6060-41b9-a442-ab840f6129e5-kube-api-access-2wjvs" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600844 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1daa8f6-7854-4c2a-b207-bd6947f3e030" volumeName="kubernetes.io/projected/e1daa8f6-7854-4c2a-b207-bd6947f3e030-kube-api-access-hrx4w" seLinuxMountContext="" Oct 29 22:15:37.600854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600857 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600869 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f585004-5478-4841-af5c-81c1c4bd9072" volumeName="kubernetes.io/projected/0f585004-5478-4841-af5c-81c1c4bd9072-kube-api-access-jnncl" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600883 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71813e11-80fa-4c04-ba56-cd1bc57d3f35" volumeName="kubernetes.io/empty-dir/71813e11-80fa-4c04-ba56-cd1bc57d3f35-volume-directive-shadow" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600895 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71813e11-80fa-4c04-ba56-cd1bc57d3f35" volumeName="kubernetes.io/projected/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-api-access-tfcc9" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600907 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c71f14d-b3bf-4942-beaa-6cdf18e331b9" volumeName="kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600923 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06ed94e-63bb-4a18-b959-4175a0972ffb" volumeName="kubernetes.io/configmap/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600936 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c71f14d-b3bf-4942-beaa-6cdf18e331b9" volumeName="kubernetes.io/projected/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-kube-api-access-cb77h" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600950 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600963 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dc91f0-4639-4294-a296-418c22a692c1" volumeName="kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-sysctl-allowlist" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600975 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71813e11-80fa-4c04-ba56-cd1bc57d3f35" volumeName="kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.600989 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/projected/82b16e7a-7a16-409f-9411-d23156de1fe1-kube-api-access-g6r2d" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601004 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601032 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601046 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601058 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f6e07a6-8957-44f5-9203-425259d84060" volumeName="kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601071 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f6e07a6-8957-44f5-9203-425259d84060" volumeName="kubernetes.io/empty-dir/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-textfile" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601084 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71813e11-80fa-4c04-ba56-cd1bc57d3f35" volumeName="kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-metrics-client-ca" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601098 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601112 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601124 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601137 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71813e11-80fa-4c04-ba56-cd1bc57d3f35" volumeName="kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601149 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f585004-5478-4841-af5c-81c1c4bd9072" volumeName="kubernetes.io/empty-dir/0f585004-5478-4841-af5c-81c1c4bd9072-tmp-dir" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601167 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="629cfb60-8afd-46e7-86a8-1de336684f20" volumeName="kubernetes.io/projected/629cfb60-8afd-46e7-86a8-1de336684f20-kube-api-access-9zrr9" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601180 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="629cfb60-8afd-46e7-86a8-1de336684f20" volumeName="kubernetes.io/secret/629cfb60-8afd-46e7-86a8-1de336684f20-ovn-node-metrics-cert" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601194 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/configmap/82b16e7a-7a16-409f-9411-d23156de1fe1-metrics-client-ca" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601206 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-tls" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601235 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601247 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dc91f0-4639-4294-a296-418c22a692c1" volumeName="kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601261 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f6e07a6-8957-44f5-9203-425259d84060" volumeName="kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-tls" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601274 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="629cfb60-8afd-46e7-86a8-1de336684f20" volumeName="kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-env-overrides" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601287 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06ed94e-63bb-4a18-b959-4175a0972ffb" volumeName="kubernetes.io/projected/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-api-access-k88ts" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601301 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a06ed94e-63bb-4a18-b959-4175a0972ffb" volumeName="kubernetes.io/secret/a06ed94e-63bb-4a18-b959-4175a0972ffb-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601313 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4a2ad5d-ac31-4ccf-8cd0-302df20209c0" volumeName="kubernetes.io/projected/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-kube-api-access-shpll" seLinuxMountContext="" Oct 29 22:15:37.601316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601327 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f585004-5478-4841-af5c-81c1c4bd9072" volumeName="kubernetes.io/secret/0f585004-5478-4841-af5c-81c1c4bd9072-metrics-tls" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601339 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601354 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601380 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601414 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601429 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601442 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f585004-5478-4841-af5c-81c1c4bd9072" volumeName="kubernetes.io/configmap/0f585004-5478-4841-af5c-81c1c4bd9072-config-volume" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601454 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f6e07a6-8957-44f5-9203-425259d84060" volumeName="kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-metrics-client-ca" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601467 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d3f4dde-95c6-41df-951b-316cd9d10f0a" volumeName="kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601480 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dc91f0-4639-4294-a296-418c22a692c1" volumeName="kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-binary-copy" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601492 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f6e07a6-8957-44f5-9203-425259d84060" volumeName="kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601505 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3b72e10-5245-4d8e-9699-bae1c061e336" volumeName="kubernetes.io/projected/b3b72e10-5245-4d8e-9699-bae1c061e336-kube-api-access-7zddq" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601518 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc7d661f-6060-41b9-a442-ab840f6129e5" volumeName="kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-daemon-config" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601532 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79" volumeName="kubernetes.io/projected/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-kube-api-access-n425l" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601547 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1daa8f6-7854-4c2a-b207-bd6947f3e030" volumeName="kubernetes.io/configmap/e1daa8f6-7854-4c2a-b207-bd6947f3e030-iptables-alerter-script" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601560 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="629cfb60-8afd-46e7-86a8-1de336684f20" volumeName="kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-script-lib" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601572 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82b16e7a-7a16-409f-9411-d23156de1fe1" volumeName="kubernetes.io/empty-dir/82b16e7a-7a16-409f-9411-d23156de1fe1-tmp" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601624 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c71f14d-b3bf-4942-beaa-6cdf18e331b9" volumeName="kubernetes.io/configmap/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-metrics-client-ca" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601637 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0efed476-7ba2-4cce-9d12-eb2e1e457160" volumeName="kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601668 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09" volumeName="kubernetes.io/configmap/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-serviceca" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601683 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa39a765-7c7f-4212-a8e2-c7f2a91f7339" volumeName="kubernetes.io/secret/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-agent-certs" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601696 5610 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79" volumeName="kubernetes.io/empty-dir/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-tmp-dir" seLinuxMountContext="" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601737 5610 reconstruct.go:97] "Volume reconstruction finished" Oct 29 22:15:37.602525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.601746 5610 reconciler.go:29] "Reconciler: start to sync state" Oct 29 22:15:37.626303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.626223 5610 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 29 22:15:37.627697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.627668 5610 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 29 22:15:37.627813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.627691 5610 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 29 22:15:37.627813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.627728 5610 kubelet.go:2521] "Starting kubelet main sync loop" Oct 29 22:15:37.631846 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:37.631798 5610 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 29 22:15:37.635533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.635511 5610 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:15:37.636105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.636086 5610 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 29 22:15:37.638429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.638409 5610 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:15:37.644966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.644938 5610 generic.go:356] "Generic (PLEG): container finished" podID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" exitCode=0 Oct 29 22:15:37.647444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.647420 5610 generic.go:356] "Generic (PLEG): container finished" podID="3f6e07a6-8957-44f5-9203-425259d84060" containerID="444db050ab90c2f66bc2da43992f7cbb38a7de80f920ffb3eb18d4586d3c09a8" exitCode=0 Oct 29 22:15:37.652782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.652758 5610 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="9964f29e5bbf6b00fdfe27a418ed875c0e595d88421958a4a1b1cf889ae6b52e" exitCode=0 Oct 29 22:15:37.652782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.652781 5610 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="822dfeffeee52e2826b1bd98f7bbaec8cde5b9e77c6a02728c1544e330c63fa7" exitCode=0 Oct 29 22:15:37.652919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.652787 5610 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="630284d39864bebfffee3bae1b5eb2baebf84ef130d3fac3ee37ad720b797ab3" exitCode=0 Oct 29 22:15:37.652919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.652793 5610 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="2443c0eec26179fd0e150f19de6466941e1d3843c44707a78276f59cb45fba4a" exitCode=0 Oct 29 22:15:37.652919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.652799 5610 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="fc800720f8eab3c5536499d83fe671e02ef9fda465235bd0b4ea21c7053b16bf" exitCode=0 Oct 29 22:15:37.652919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.652804 5610 generic.go:356] "Generic (PLEG): container finished" podID="35dc91f0-4639-4294-a296-418c22a692c1" containerID="afc510cc36cdd11a57e8d8d7e2dd40908806042ddad3463e285bccfba45518f0" exitCode=0 Oct 29 22:15:37.655316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.655296 5610 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:15:37.655446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.655316 5610 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 29 22:15:37.684085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.684045 5610 generic.go:356] "Generic (PLEG): container finished" podID="150171fc6f62800f31d26bdb941c8f1d" containerID="f2a86176bd78be0bacc7b2a4f21ce053b71ccfc2d68d443c11ef4f6d5569f9de" exitCode=0 Oct 29 22:15:37.712339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.712319 5610 manager.go:324] Recovery completed Oct 29 22:15:37.732035 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:37.731966 5610 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 29 22:15:37.733546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733520 5610 cpu_manager.go:226] "Starting" policy="none" Oct 29 22:15:37.733546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733537 5610 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 29 22:15:37.733702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733557 5610 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 29 22:15:37.733764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733743 5610 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 29 22:15:37.733814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733759 5610 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 29 22:15:37.733814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733784 5610 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 29 22:15:37.733814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733792 5610 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 29 22:15:37.733814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733800 5610 policy_none.go:50] "Start" Oct 29 22:15:37.733814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733812 5610 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 29 22:15:37.734041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.733826 5610 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 29 22:15:37.734041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.734007 5610 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 29 22:15:37.734041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.734019 5610 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 29 22:15:37.734041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.734027 5610 policy_none.go:44] "Start" Oct 29 22:15:37.738413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.738374 5610 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 29 22:15:37.743419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.743383 5610 manager.go:341] "Starting Device Plugin manager" Oct 29 22:15:37.743863 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:37.743431 5610 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 29 22:15:37.743916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.743867 5610 server.go:87] "Starting device plugin registration server" Oct 29 22:15:37.744124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.744114 5610 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 29 22:15:37.744160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.744125 5610 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 29 22:15:37.744250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.744228 5610 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 29 22:15:37.744374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.744337 5610 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 29 22:15:37.744466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.744380 5610 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 29 22:15:37.744466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.744387 5610 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 29 22:15:37.744555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.744434 5610 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:15:37.744369707Z","UUID":"8daac0fe-56e4-4130-ac7b-e0bcfd12d20e","Handler":null,"Name":"","Endpoint":""} Oct 29 22:15:37.745868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.745852 5610 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 29 22:15:37.745976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.745878 5610 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 29 22:15:37.746295 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:37.746275 5610 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 29 22:15:37.845251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.845221 5610 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 29 22:15:37.848322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.848300 5610 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory" Oct 29 22:15:37.848457 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.848339 5610 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure" Oct 29 22:15:37.848457 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.848353 5610 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID" Oct 29 22:15:37.848457 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.848418 5610 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-251.ec2.internal" Oct 29 22:15:37.878727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.878702 5610 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-251.ec2.internal" Oct 29 22:15:37.878815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.878773 5610 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-251.ec2.internal" Oct 29 22:15:37.932532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.932452 5610 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"] Oct 29 22:15:37.933081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.933043 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" event={"ID":"df1fead76acc43615f3bffca1a53f41e","Type":"ContainerStarted","Data":"70e4f9a655b73db079b40320ea7478de2be4a0e224b7a23abb2f91042360e091"} Oct 29 22:15:37.933206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.933091 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" event={"ID":"df1fead76acc43615f3bffca1a53f41e","Type":"ContainerStarted","Data":"7931779220d01a3c9e6706d456b9431572ac5cb47009367ae9fb0f5c43541e8a"} Oct 29 22:15:37.933206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.933113 5610 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:37.933206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.933164 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"150171fc6f62800f31d26bdb941c8f1d","Type":"ContainerStarted","Data":"616acac671d6c5f7f9b09e479b731bab783800a4cd8780ad90aeb11876e8d55a"} Oct 29 22:15:37.933206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.933185 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"150171fc6f62800f31d26bdb941c8f1d","Type":"ContainerDied","Data":"f2a86176bd78be0bacc7b2a4f21ce053b71ccfc2d68d443c11ef4f6d5569f9de"} Oct 29 22:15:37.933206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.933204 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"150171fc6f62800f31d26bdb941c8f1d","Type":"ContainerStarted","Data":"7eee30eb857c093ae68fcc631f38870418c1ea438d4d56ea3e50225dac2c09dd"} Oct 29 22:15:37.933471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.933287 5610 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:15:37.952921 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.952898 5610 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:15:37.953005 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:37.952950 5610 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:15:37.953345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:37.953331 5610 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:15:37.953377 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:37.953365 5610 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.038129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.038038 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.038129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.038109 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.038129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.038129 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df1fead76acc43615f3bffca1a53f41e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"df1fead76acc43615f3bffca1a53f41e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.138683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.138649 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.138683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.138684 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.138851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.138708 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df1fead76acc43615f3bffca1a53f41e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"df1fead76acc43615f3bffca1a53f41e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.138851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.138757 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.138851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.138760 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df1fead76acc43615f3bffca1a53f41e-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"df1fead76acc43615f3bffca1a53f41e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.138851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.138761 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/150171fc6f62800f31d26bdb941c8f1d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"150171fc6f62800f31d26bdb941c8f1d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.583647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.583611 5610 apiserver.go:52] "Watching apiserver" Oct 29 22:15:38.626541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.626518 5610 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 29 22:15:38.627106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.627086 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5trph","openshift-monitoring/thanos-querier-848cccd68d-hlbqr","openshift-multus/multus-5rqc2","openshift-multus/multus-additional-cni-plugins-dkbgr","openshift-network-operator/iptables-alerter-84hqt","openshift-cluster-node-tuning-operator/tuned-x8plf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal","openshift-dns/dns-default-6sljl","openshift-insights/insights-runtime-extractor-446fn","openshift-dns/node-resolver-lmmsc","openshift-ovn-kubernetes/ovnkube-node-zflnw","openshift-network-diagnostics/network-check-target-qq6r4","openshift-monitoring/node-exporter-424dq","openshift-ingress-canary/ingress-canary-2vcms","openshift-multus/network-metrics-daemon-nmlxs","openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc","kube-system/konnectivity-agent-6v7fk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m","openshift-image-registry/node-ca-vhtpp","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk","openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs","kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"] Oct 29 22:15:38.645445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.645419 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 29 22:15:38.645588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.645547 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 29 22:15:38.645672 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.645659 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.645792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.645779 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.645822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.645800 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 29 22:15:38.645938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.645926 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.645988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.645935 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 29 22:15:38.646065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.646048 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.646144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.646061 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.646144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.646071 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.653272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.653253 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 29 22:15:38.653782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.653765 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.654092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654075 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 29 22:15:38.654157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654087 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 29 22:15:38.654157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654079 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 29 22:15:38.654477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654449 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 29 22:15:38.654595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654577 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gbljw\"" Oct 29 22:15:38.654682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654581 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 29 22:15:38.654682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654601 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2cw6s\"" Oct 29 22:15:38.654682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654581 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 29 22:15:38.654682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654635 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654736 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pxx6w\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654744 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654770 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654774 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kk6hd\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654796 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfkqw\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654806 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-475gp\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654823 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w24c4\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654826 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hznbv\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654886 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4fh5h\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654888 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm48d\"" Oct 29 22:15:38.654944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.654940 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dk2rc\"" Oct 29 22:15:38.655341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.655079 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t5dlz\"" Oct 29 22:15:38.655341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.655092 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 29 22:15:38.655341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.655102 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cbfws\"" Oct 29 22:15:38.655341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.655114 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-h6g72\"" Oct 29 22:15:38.655341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.655132 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 29 22:15:38.657279 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.657266 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 29 22:15:38.658439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.658424 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qnnrl\"" Oct 29 22:15:38.658498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.658475 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 29 22:15:38.658557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.658533 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 29 22:15:38.660492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.660477 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 29 22:15:38.660558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.660493 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tqdrf\"" Oct 29 22:15:38.663225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.663212 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 29 22:15:38.663302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.663291 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 29 22:15:38.663728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.663704 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 22:15:38.664609 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.664593 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 22:15:38.664709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.664596 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-28glh\"" Oct 29 22:15:38.664709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.664629 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r9x6m\"" Oct 29 22:15:38.664709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.664635 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-csh82vmno0a66\"" Oct 29 22:15:38.664709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.664640 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 29 22:15:38.664709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.664684 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 29 22:15:38.664709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.664687 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 29 22:15:38.664958 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.664907 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-656vh\"" Oct 29 22:15:38.666917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.666900 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7kpma9rsabkp1\"" Oct 29 22:15:38.666982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.666935 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 29 22:15:38.666982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.666962 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 29 22:15:38.667075 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667017 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 22:15:38.667122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667107 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 22:15:38.667479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667462 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 29 22:15:38.667558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667524 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 22:15:38.667558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667524 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 22:15:38.667558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667529 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 22:15:38.667558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667548 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 22:15:38.667558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667552 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 22:15:38.667736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667568 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 22:15:38.667736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667664 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kptl2\"" Oct 29 22:15:38.667736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667717 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.667736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667710 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 22:15:38.667948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667764 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.667948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667772 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.667948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667785 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.667948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667797 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.667948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667823 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 29 22:15:38.667948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667829 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.667948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.667867 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 29 22:15:38.668223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668203 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.668223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668205 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 29 22:15:38.668345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668333 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.668469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668455 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.668529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668488 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hg8d9\"" Oct 29 22:15:38.668529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668492 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.668529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668513 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.668529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668513 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.668707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668538 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 22:15:38.668707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668599 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 29 22:15:38.668707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668677 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.668707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668687 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 29 22:15:38.668875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668708 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 29 22:15:38.668875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668745 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 29 22:15:38.668875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668744 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 22:15:38.668875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668764 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 29 22:15:38.668875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.668800 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 29 22:15:38.674585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.674563 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 22:15:38.674759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.674743 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 22:15:38.694581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.694561 5610 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.694708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.694696 5610 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.700561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.700546 5610 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 29 22:15:38.712444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.712431 5610 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:15:38.712492 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:38.712474 5610 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.712870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.712858 5610 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 29 22:15:38.712900 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:38.712891 5610 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" Oct 29 22:15:38.742603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742579 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.742603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742606 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.742765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742624 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.742765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742642 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.742765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742664 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.742902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742767 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.742902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742805 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-node-log\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.742902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742848 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.742902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742879 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.743049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742912 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.743049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742916 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.743049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742882 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.743049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.742914 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.743187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743063 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-netns\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.743187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743078 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.743187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743086 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-bin\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.743187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743102 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-systemd\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.743187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743120 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-device-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.743187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743157 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-textfile\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.743187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743176 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.743493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743221 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-textfile\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.743493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743225 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-cnibin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.743493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743316 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-host\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:15:38.743493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743353 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysconfig\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.743493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743379 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-konnectivity-ca\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:15:38.743493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743426 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrx4w\" (UniqueName: \"kubernetes.io/projected/e1daa8f6-7854-4c2a-b207-bd6947f3e030-kube-api-access-hrx4w\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:15:38.743493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743476 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.743720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743590 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-tls\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.743720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743605 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-konnectivity-ca\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:15:38.743720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743629 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.743857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743753 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-systemd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.743857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743827 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-tls\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.743857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743832 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.743970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743903 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8769\" (UniqueName: \"kubernetes.io/projected/35dc91f0-4639-4294-a296-418c22a692c1-kube-api-access-d8769\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.743970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743939 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b731a32-138f-4eb8-830a-7078d3667e68-cert\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:38.743970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743953 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.743970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.743965 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-hostroot\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.744145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744086 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:38.744197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744162 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:38.744240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744198 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-modprobe-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.744240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744215 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b731a32-138f-4eb8-830a-7078d3667e68-cert\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:38.744240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744221 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.744370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744244 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.744370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744275 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1daa8f6-7854-4c2a-b207-bd6947f3e030-iptables-alerter-script\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:15:38.744370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744304 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.744370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744336 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88ts\" (UniqueName: \"kubernetes.io/projected/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-api-access-k88ts\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.744370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744366 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744440 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744466 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab618391-4607-4f00-ab02-819dc9897da1-original-pull-secret\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744481 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27m6m\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744515 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744541 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-cnibin\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744560 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744569 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1daa8f6-7854-4c2a-b207-bd6947f3e030-iptables-alerter-script\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744595 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:38.744630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744625 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71813e11-80fa-4c04-ba56-cd1bc57d3f35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744653 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-system-cni-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744673 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-metrics-client-ca\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744714 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/71813e11-80fa-4c04-ba56-cd1bc57d3f35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744728 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744748 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744760 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744774 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n425l\" (UniqueName: \"kubernetes.io/projected/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-kube-api-access-n425l\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744801 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-slash\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744825 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8md\" (UniqueName: \"kubernetes.io/projected/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-kube-api-access-sd8md\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744848 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-host\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744867 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744888 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-metrics-client-ca\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744911 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-tuned\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.745032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.744939 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745046 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-etc-selinux\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745073 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745090 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-hosts-file\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745090 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-tuned\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745117 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745149 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745201 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745277 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-run\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745301 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-var-lib-kubelet\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745306 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745317 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-registration-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745352 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745364 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b16e7a-7a16-409f-9411-d23156de1fe1-tmp\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745385 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745426 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnncl\" (UniqueName: \"kubernetes.io/projected/0f585004-5478-4841-af5c-81c1c4bd9072-kube-api-access-jnncl\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745459 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b16e7a-7a16-409f-9411-d23156de1fe1-tmp\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745463 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-lib-modules\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745520 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-accelerators-collector-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745570 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745679 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745717 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745729 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-accelerators-collector-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.745734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745738 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745816 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745837 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745848 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-kubernetes\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745882 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zddq\" (UniqueName: \"kubernetes.io/projected/b3b72e10-5245-4d8e-9699-bae1c061e336-kube-api-access-7zddq\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745913 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtsh\" (UniqueName: \"kubernetes.io/projected/3b731a32-138f-4eb8-830a-7078d3667e68-kube-api-access-6dtsh\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745951 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6r2d\" (UniqueName: \"kubernetes.io/projected/82b16e7a-7a16-409f-9411-d23156de1fe1-kube-api-access-g6r2d\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.745971 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn28n\" (UniqueName: \"kubernetes.io/projected/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-kube-api-access-wn28n\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746022 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746088 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746123 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746149 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746183 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-systemd-units\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746211 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrr9\" (UniqueName: \"kubernetes.io/projected/629cfb60-8afd-46e7-86a8-1de336684f20-kube-api-access-9zrr9\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746244 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-tmp\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746302 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfd4dd9-a2a6-4ecd-9667-840738787716-tmp\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746326 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746343 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746367 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-kubelet\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746410 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-socket-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746448 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746476 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746518 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1daa8f6-7854-4c2a-b207-bd6947f3e030-host-slash\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746544 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-var-lib-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746629 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.746718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746719 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-kubelet-config\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746738 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-conf\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746755 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-cni-binary-copy\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746752 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746771 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-netns\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746815 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-etc-kubernetes\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746850 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f585004-5478-4841-af5c-81c1c4bd9072-config-volume\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746898 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746940 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.746964 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-cni-binary-copy\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747008 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-bin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747079 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f585004-5478-4841-af5c-81c1c4bd9072-config-volume\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747092 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747113 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-ovn\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747171 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f585004-5478-4841-af5c-81c1c4bd9072-tmp-dir\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747183 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747198 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a06ed94e-63bb-4a18-b959-4175a0972ffb-data-volume\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747218 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82b16e7a-7a16-409f-9411-d23156de1fe1-metrics-client-ca\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747233 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747260 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f585004-5478-4841-af5c-81c1c4bd9072-tmp-dir\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747265 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfcc9\" (UniqueName: \"kubernetes.io/projected/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-api-access-tfcc9\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747287 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-script-lib\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747306 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shpll\" (UniqueName: \"kubernetes.io/projected/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-kube-api-access-shpll\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747312 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a06ed94e-63bb-4a18-b959-4175a0972ffb-data-volume\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747321 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb77h\" (UniqueName: \"kubernetes.io/projected/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-kube-api-access-cb77h\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747337 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-root\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747352 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747367 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-sys-fs\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747381 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-conf-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747421 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-multus-certs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747441 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747456 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-sys\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747471 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-sys\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747487 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747504 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747522 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-tmp-dir\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747544 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-kubelet\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.747550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747551 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82b16e7a-7a16-409f-9411-d23156de1fe1-metrics-client-ca\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747627 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747703 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747707 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747758 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-script-lib\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747787 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-tmp-dir\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747819 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747826 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747857 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-etc-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747880 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747953 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-metrics-certs\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.747989 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-grpc-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748010 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748017 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-wtmp\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748053 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748075 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-system-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748100 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-k8s-cni-cncf-io\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748119 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748124 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-netd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748165 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-log-socket\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748195 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-config\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748236 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748251 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-agent-certs\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748260 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-grpc-tls\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748278 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8m5\" (UniqueName: \"kubernetes.io/projected/4bfd4dd9-a2a6-4ecd-9667-840738787716-kube-api-access-pj8m5\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748311 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748339 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-multus\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748356 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-ovnkube-config\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748363 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-socket-dir-parent\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748501 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-daemon-config\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748519 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748534 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/629cfb60-8afd-46e7-86a8-1de336684f20-ovn-node-metrics-cert\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.748555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748535 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa39a765-7c7f-4212-a8e2-c7f2a91f7339-agent-certs\") pod \"konnectivity-agent-6v7fk\" (UID: \"aa39a765-7c7f-4212-a8e2-c7f2a91f7339\") " pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748618 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748727 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-daemon-config\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748750 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a06ed94e-63bb-4a18-b959-4175a0972ffb-crio-socket\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748761 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748769 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a06ed94e-63bb-4a18-b959-4175a0972ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748884 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748904 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/629cfb60-8afd-46e7-86a8-1de336684f20-ovn-node-metrics-cert\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748911 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjvs\" (UniqueName: \"kubernetes.io/projected/cc7d661f-6060-41b9-a442-ab840f6129e5-kube-api-access-2wjvs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.748929 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749001 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a06ed94e-63bb-4a18-b959-4175a0972ffb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749014 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-env-overrides\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749044 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82b16e7a-7a16-409f-9411-d23156de1fe1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749047 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749073 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-serviceca\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749092 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-dbus\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749132 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f585004-5478-4841-af5c-81c1c4bd9072-metrics-tls\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749195 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/629cfb60-8afd-46e7-86a8-1de336684f20-env-overrides\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749217 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqh2\" (UniqueName: \"kubernetes.io/projected/3f6e07a6-8957-44f5-9203-425259d84060-kube-api-access-mwqh2\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749319 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35dc91f0-4639-4294-a296-418c22a692c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749364 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f585004-5478-4841-af5c-81c1c4bd9072-metrics-tls\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:38.749521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.749504 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-serviceca\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:15:38.759710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.759687 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8769\" (UniqueName: \"kubernetes.io/projected/35dc91f0-4639-4294-a296-418c22a692c1-kube-api-access-d8769\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.759807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.759725 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrx4w\" (UniqueName: \"kubernetes.io/projected/e1daa8f6-7854-4c2a-b207-bd6947f3e030-kube-api-access-hrx4w\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:15:38.759845 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.759804 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8md\" (UniqueName: \"kubernetes.io/projected/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-kube-api-access-sd8md\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:15:38.759885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.759870 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88ts\" (UniqueName: \"kubernetes.io/projected/a06ed94e-63bb-4a18-b959-4175a0972ffb-kube-api-access-k88ts\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.760590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.760571 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n425l\" (UniqueName: \"kubernetes.io/projected/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-kube-api-access-n425l\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:15:38.761052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.761027 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6ww\" (UniqueName: \"kubernetes.io/projected/1d3f4dde-95c6-41df-951b-316cd9d10f0a-kube-api-access-zn6ww\") pod \"network-check-target-qq6r4\" (UID: \"1d3f4dde-95c6-41df-951b-316cd9d10f0a\") " pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:38.761154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.761107 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27m6m\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m\") pod \"prometheus-k8s-0\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:38.761216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.761188 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnncl\" (UniqueName: \"kubernetes.io/projected/0f585004-5478-4841-af5c-81c1c4bd9072-kube-api-access-jnncl\") pod \"dns-default-6sljl\" (UID: \"0f585004-5478-4841-af5c-81c1c4bd9072\") " pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:38.761380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.761363 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtsh\" (UniqueName: \"kubernetes.io/projected/3b731a32-138f-4eb8-830a-7078d3667e68-kube-api-access-6dtsh\") pod \"ingress-canary-2vcms\" (UID: \"3b731a32-138f-4eb8-830a-7078d3667e68\") " pod="openshift-ingress-canary/ingress-canary-2vcms" Oct 29 22:15:38.761498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.761481 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrr9\" (UniqueName: \"kubernetes.io/projected/629cfb60-8afd-46e7-86a8-1de336684f20-kube-api-access-9zrr9\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.761791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.761773 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6r2d\" (UniqueName: \"kubernetes.io/projected/82b16e7a-7a16-409f-9411-d23156de1fe1-kube-api-access-g6r2d\") pod \"thanos-querier-848cccd68d-hlbqr\" (UID: \"82b16e7a-7a16-409f-9411-d23156de1fe1\") " pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:38.762000 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.761984 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zddq\" (UniqueName: \"kubernetes.io/projected/b3b72e10-5245-4d8e-9699-bae1c061e336-kube-api-access-7zddq\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.762060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.762041 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn28n\" (UniqueName: \"kubernetes.io/projected/d0c9a67d-21dc-44cf-ab1f-1ef20fba70af-kube-api-access-wn28n\") pod \"monitoring-plugin-77f4f76594-vn7xs\" (UID: \"d0c9a67d-21dc-44cf-ab1f-1ef20fba70af\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:38.762690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.762673 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfcc9\" (UniqueName: \"kubernetes.io/projected/71813e11-80fa-4c04-ba56-cd1bc57d3f35-kube-api-access-tfcc9\") pod \"kube-state-metrics-5dc8d7d89b-5thtk\" (UID: \"71813e11-80fa-4c04-ba56-cd1bc57d3f35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5thtk" Oct 29 22:15:38.762776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.762701 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shpll\" (UniqueName: \"kubernetes.io/projected/b4a2ad5d-ac31-4ccf-8cd0-302df20209c0-kube-api-access-shpll\") pod \"network-metrics-daemon-nmlxs\" (UID: \"b4a2ad5d-ac31-4ccf-8cd0-302df20209c0\") " pod="openshift-multus/network-metrics-daemon-nmlxs" Oct 29 22:15:38.763663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.763642 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjvs\" (UniqueName: \"kubernetes.io/projected/cc7d661f-6060-41b9-a442-ab840f6129e5-kube-api-access-2wjvs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.763736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.763644 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8m5\" (UniqueName: \"kubernetes.io/projected/4bfd4dd9-a2a6-4ecd-9667-840738787716-kube-api-access-pj8m5\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.763917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.763902 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb77h\" (UniqueName: \"kubernetes.io/projected/9c71f14d-b3bf-4942-beaa-6cdf18e331b9-kube-api-access-cb77h\") pod \"openshift-state-metrics-7bd7489957-xc2mc\" (UID: \"9c71f14d-b3bf-4942-beaa-6cdf18e331b9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xc2mc" Oct 29 22:15:38.764144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.764127 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqh2\" (UniqueName: \"kubernetes.io/projected/3f6e07a6-8957-44f5-9203-425259d84060-kube-api-access-mwqh2\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.850331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850254 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-node-log\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850286 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.850331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850303 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-netns\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850317 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-bin\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850332 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-systemd\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850346 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-device-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850365 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-cnibin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850386 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-host\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850420 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-netns\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850447 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-cnibin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850454 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09-host\") pod \"node-ca-vhtpp\" (UID: \"3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09\") " pod="openshift-image-registry/node-ca-vhtpp" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850420 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-bin\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850473 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysconfig\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850429 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysconfig\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850383 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850446 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-systemd\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850383 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-node-log\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850494 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-device-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850518 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850541 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-systemd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850557 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850572 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850575 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-hostroot\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850610 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-systemd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.850629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850618 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-hostroot\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850646 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850664 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-modprobe-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850685 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850702 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850717 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-cnibin\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850735 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-modprobe-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850740 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-system-cni-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850763 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-run-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850774 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-system-cni-dir\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850768 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35dc91f0-4639-4294-a296-418c22a692c1-cnibin\") pod \"multus-additional-cni-plugins-dkbgr\" (UID: \"35dc91f0-4639-4294-a296-418c22a692c1\") " pod="openshift-multus/multus-additional-cni-plugins-dkbgr" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850779 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-slash\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850797 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-slash\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850751 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850808 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-host\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850823 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-etc-selinux\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850839 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-hosts-file\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850847 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-host\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850855 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-run\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850869 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-var-lib-kubelet\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850883 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-registration-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850913 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-lib-modules\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850941 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-kubernetes\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850945 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-run\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850950 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-var-lib-kubelet\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850938 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79-hosts-file\") pod \"node-resolver-lmmsc\" (UID: \"d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79\") " pod="openshift-dns/node-resolver-lmmsc" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850974 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-etc-selinux\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850985 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-systemd-units\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850993 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-kubernetes\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.850988 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-registration-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851011 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-kubelet\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851018 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-systemd-units\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851026 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-socket-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851054 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-lib-modules\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851065 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-socket-dir\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851055 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1daa8f6-7854-4c2a-b207-bd6947f3e030-host-slash\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851081 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-kubelet\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851095 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-var-lib-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851118 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-kubelet-config\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851132 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-conf\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851137 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-var-lib-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851150 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-netns\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851084 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1daa8f6-7854-4c2a-b207-bd6947f3e030-host-slash\") pod \"iptables-alerter-84hqt\" (UID: \"e1daa8f6-7854-4c2a-b207-bd6947f3e030\") " pod="openshift-network-operator/iptables-alerter-84hqt" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851173 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-etc-kubernetes\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851189 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-kubelet-config\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851194 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-netns\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851200 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-bin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851222 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-bin\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851225 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-ovn\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851226 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-etc-kubernetes\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851253 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-root\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851273 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851254 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-run-ovn\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.851297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851327 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-root\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851245 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-conf\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851371 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-etc-sysctl-d\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851426 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-sys-fs\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851288 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3b72e10-5245-4d8e-9699-bae1c061e336-sys-fs\") pod \"aws-ebs-csi-driver-node-zq84m\" (UID: \"b3b72e10-5245-4d8e-9699-bae1c061e336\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zq84m" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851729 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-conf-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851744 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-multus-certs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851758 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-sys\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851772 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-sys\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851789 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-kubelet\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851810 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-etc-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851820 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-conf-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851834 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-wtmp\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851844 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-sys\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851858 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-system-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851871 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-etc-openvswitch\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851881 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-k8s-cni-cncf-io\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851883 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfd4dd9-a2a6-4ecd-9667-840738787716-sys\") pod \"tuned-x8plf\" (UID: \"4bfd4dd9-a2a6-4ecd-9667-840738787716\") " pod="openshift-cluster-node-tuning-operator/tuned-x8plf" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851848 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-kubelet\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851859 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-multus-certs\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851913 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-netd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851918 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-system-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851938 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f6e07a6-8957-44f5-9203-425259d84060-node-exporter-wtmp\") pod \"node-exporter-424dq\" (UID: \"3f6e07a6-8957-44f5-9203-425259d84060\") " pod="openshift-monitoring/node-exporter-424dq" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851941 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-run-k8s-cni-cncf-io\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851940 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-host-cni-netd\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851966 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-log-socket\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851939 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/629cfb60-8afd-46e7-86a8-1de336684f20-log-socket\") pod \"ovnkube-node-zflnw\" (UID: \"629cfb60-8afd-46e7-86a8-1de336684f20\") " pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.851997 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-multus\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852012 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-socket-dir-parent\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852029 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a06ed94e-63bb-4a18-b959-4175a0972ffb-crio-socket\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852043 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852061 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-dbus\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852092 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-socket-dir-parent\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852110 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a06ed94e-63bb-4a18-b959-4175a0972ffb-crio-socket\") pod \"insights-runtime-extractor-446fn\" (UID: \"a06ed94e-63bb-4a18-b959-4175a0972ffb\") " pod="openshift-insights/insights-runtime-extractor-446fn" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852117 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-host-var-lib-cni-multus\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852134 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab618391-4607-4f00-ab02-819dc9897da1-dbus\") pod \"global-pull-secret-syncer-5trph\" (UID: \"ab618391-4607-4f00-ab02-819dc9897da1\") " pod="kube-system/global-pull-secret-syncer-5trph" Oct 29 22:15:38.852807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:38.852143 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc7d661f-6060-41b9-a442-ab840f6129e5-multus-cni-dir\") pod \"multus-5rqc2\" (UID: \"cc7d661f-6060-41b9-a442-ab840f6129e5\") " pod="openshift-multus/multus-5rqc2" Oct 29 22:15:39.741494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:39.741462 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6sljl" Oct 29 22:15:40.822370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:40.822335 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:15:41.021731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:41.021693 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:15:41.087503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:41.087380 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:41.228447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:41.228389 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:41.268955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:41.268927 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:42.086359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:42.086323 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" Oct 29 22:15:42.463531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:42.463502 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-848cccd68d-hlbqr" Oct 29 22:15:43.263558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:43.263534 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:43.263954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:43.263719 5610 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 29 22:15:43.707326 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:15:43.707288 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:15:44.927619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:44.927576 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:45.063185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:45.063143 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:46.730328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:46.730302 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6v7fk" Oct 29 22:15:51.022464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:51.022369 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:51.147547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:51.147515 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:52.965123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:52.965095 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zflnw" Oct 29 22:15:54.068165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:54.068129 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:54.238976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:54.238934 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:59.932143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:59.932107 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:15:59.960532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:59.960498 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:15:59.961127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:59.961063 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy" containerID="cri-o://06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" gracePeriod=600 Oct 29 22:15:59.961127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:59.961118 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" gracePeriod=600 Oct 29 22:15:59.961359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:59.961117 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="config-reloader" containerID="cri-o://0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" gracePeriod=600 Oct 29 22:15:59.961359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:59.961117 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy-web" containerID="cri-o://f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" gracePeriod=600 Oct 29 22:15:59.961359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:59.961044 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="prometheus" containerID="cri-o://1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" gracePeriod=600 Oct 29 22:15:59.961359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:15:59.961077 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="thanos-sidecar" containerID="cri-o://ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" gracePeriod=600 Oct 29 22:16:00.068917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.068874 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:00.197451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.197430 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.232878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.232848 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.232882 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.232910 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.232940 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233123 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233178 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233208 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233237 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233267 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233308 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233337 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:00.233364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233350 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233379 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233856 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233908 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.233961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233940 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.234064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.233976 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.234064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.234026 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.234170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.234064 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.234170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.234104 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.234170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.234136 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file\") pod \"0efed476-7ba2-4cce-9d12-eb2e1e457160\" (UID: \"0efed476-7ba2-4cce-9d12-eb2e1e457160\") " Oct 29 22:16:00.234381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.234362 5610 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.234472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.234413 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:00.235362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.235062 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:00.235362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.235104 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:00.236108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.236082 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:00.236491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.236467 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:00.236570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.236510 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:00.237074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.237051 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:00.237207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.237188 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "kube-api-access-27m6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:16:00.237647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.237619 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.237817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.237786 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.237817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.237808 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.237999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.237963 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.238120 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.238104 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.238208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.238191 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:16:00.238270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.238218 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:16:00.238360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.238337 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.238435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.238383 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.239495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.239471 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.246927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.246902 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config" pod "0efed476-7ba2-4cce-9d12-eb2e1e457160" (UID: "0efed476-7ba2-4cce-9d12-eb2e1e457160"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:16:00.335110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335076 5610 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-tls-assets\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335108 5610 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335122 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27m6m\" (UniqueName: \"kubernetes.io/projected/0efed476-7ba2-4cce-9d12-eb2e1e457160-kube-api-access-27m6m\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335136 5610 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-config-out\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335147 5610 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-db\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335158 5610 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-metrics-client-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335171 5610 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335185 5610 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335197 5610 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335210 5610 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335222 5610 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-web-config\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335234 5610 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335248 5610 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-grpc-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335260 5610 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335272 5610 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-config\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335284 5610 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-configmap-metrics-client-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335297 5610 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-secret-kube-rbac-proxy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335310 5610 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0efed476-7ba2-4cce-9d12-eb2e1e457160-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.335350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.335324 5610 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0efed476-7ba2-4cce-9d12-eb2e1e457160-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:16:00.774183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774146 5610 generic.go:356] "Generic (PLEG): container finished" podID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" exitCode=0 Oct 29 22:16:00.774183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774172 5610 generic.go:356] "Generic (PLEG): container finished" podID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" exitCode=0 Oct 29 22:16:00.774183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774181 5610 generic.go:356] "Generic (PLEG): container finished" podID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" exitCode=0 Oct 29 22:16:00.774183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774188 5610 generic.go:356] "Generic (PLEG): container finished" podID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" exitCode=0 Oct 29 22:16:00.774183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774194 5610 generic.go:356] "Generic (PLEG): container finished" podID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" exitCode=0 Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774199 5610 generic.go:356] "Generic (PLEG): container finished" podID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" exitCode=0 Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774221 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerDied","Data":"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627"} Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774238 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774289 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerDied","Data":"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81"} Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774316 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerDied","Data":"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995"} Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774341 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerDied","Data":"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672"} Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774353 5610 scope.go:122] "RemoveContainer" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774366 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerDied","Data":"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100"} Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774401 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerDied","Data":"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e"} Oct 29 22:16:00.774618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.774428 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0efed476-7ba2-4cce-9d12-eb2e1e457160","Type":"ContainerDied","Data":"495cd647124ee7f1bb17ba294b19464e4ae192eeafa4b1623c98792cb046bcc4"} Oct 29 22:16:00.781961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.781943 5610 scope.go:122] "RemoveContainer" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" Oct 29 22:16:00.788986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.788967 5610 scope.go:122] "RemoveContainer" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" Oct 29 22:16:00.795539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.795519 5610 scope.go:122] "RemoveContainer" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" Oct 29 22:16:00.801823 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.801802 5610 scope.go:122] "RemoveContainer" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" Oct 29 22:16:00.808366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.808347 5610 scope.go:122] "RemoveContainer" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" Oct 29 22:16:00.815284 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.815267 5610 scope.go:122] "RemoveContainer" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" Oct 29 22:16:00.821662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.821645 5610 scope.go:122] "RemoveContainer" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" Oct 29 22:16:00.821957 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:00.821937 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": container with ID starting with 7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627 not found: ID does not exist" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" Oct 29 22:16:00.822024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.821970 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627"} err="failed to get container status \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": rpc error: code = NotFound desc = could not find container \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": container with ID starting with 7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627 not found: ID does not exist" Oct 29 22:16:00.822024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.822015 5610 scope.go:122] "RemoveContainer" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" Oct 29 22:16:00.822297 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:00.822280 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": container with ID starting with 06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81 not found: ID does not exist" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" Oct 29 22:16:00.822337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.822304 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81"} err="failed to get container status \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": rpc error: code = NotFound desc = could not find container \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": container with ID starting with 06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81 not found: ID does not exist" Oct 29 22:16:00.822337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.822320 5610 scope.go:122] "RemoveContainer" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" Oct 29 22:16:00.822641 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:00.822626 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": container with ID starting with f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995 not found: ID does not exist" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" Oct 29 22:16:00.822707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.822650 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995"} err="failed to get container status \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": rpc error: code = NotFound desc = could not find container \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": container with ID starting with f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995 not found: ID does not exist" Oct 29 22:16:00.822707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.822672 5610 scope.go:122] "RemoveContainer" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" Oct 29 22:16:00.822885 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:00.822869 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": container with ID starting with ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672 not found: ID does not exist" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" Oct 29 22:16:00.822922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.822890 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672"} err="failed to get container status \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": rpc error: code = NotFound desc = could not find container \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": container with ID starting with ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672 not found: ID does not exist" Oct 29 22:16:00.822922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.822903 5610 scope.go:122] "RemoveContainer" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" Oct 29 22:16:00.823137 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:00.823119 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": container with ID starting with 0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100 not found: ID does not exist" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" Oct 29 22:16:00.823203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.823146 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100"} err="failed to get container status \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": rpc error: code = NotFound desc = could not find container \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": container with ID starting with 0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100 not found: ID does not exist" Oct 29 22:16:00.823203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.823163 5610 scope.go:122] "RemoveContainer" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" Oct 29 22:16:00.823380 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:00.823362 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": container with ID starting with 1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e not found: ID does not exist" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" Oct 29 22:16:00.823441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.823383 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e"} err="failed to get container status \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": rpc error: code = NotFound desc = could not find container \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": container with ID starting with 1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e not found: ID does not exist" Oct 29 22:16:00.823441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.823409 5610 scope.go:122] "RemoveContainer" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" Oct 29 22:16:00.823637 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:00.823620 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": container with ID starting with c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436 not found: ID does not exist" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" Oct 29 22:16:00.823677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.823651 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436"} err="failed to get container status \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": rpc error: code = NotFound desc = could not find container \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": container with ID starting with c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436 not found: ID does not exist" Oct 29 22:16:00.823677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.823664 5610 scope.go:122] "RemoveContainer" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" Oct 29 22:16:00.823881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.823864 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627"} err="failed to get container status \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": rpc error: code = NotFound desc = could not find container \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": container with ID starting with 7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627 not found: ID does not exist" Oct 29 22:16:00.823922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.823882 5610 scope.go:122] "RemoveContainer" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" Oct 29 22:16:00.824059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824041 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81"} err="failed to get container status \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": rpc error: code = NotFound desc = could not find container \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": container with ID starting with 06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81 not found: ID does not exist" Oct 29 22:16:00.824099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824060 5610 scope.go:122] "RemoveContainer" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" Oct 29 22:16:00.824220 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824201 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995"} err="failed to get container status \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": rpc error: code = NotFound desc = could not find container \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": container with ID starting with f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995 not found: ID does not exist" Oct 29 22:16:00.824286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824220 5610 scope.go:122] "RemoveContainer" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" Oct 29 22:16:00.824482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824463 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672"} err="failed to get container status \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": rpc error: code = NotFound desc = could not find container \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": container with ID starting with ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672 not found: ID does not exist" Oct 29 22:16:00.824526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824483 5610 scope.go:122] "RemoveContainer" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" Oct 29 22:16:00.824696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824678 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100"} err="failed to get container status \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": rpc error: code = NotFound desc = could not find container \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": container with ID starting with 0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100 not found: ID does not exist" Oct 29 22:16:00.824762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824698 5610 scope.go:122] "RemoveContainer" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" Oct 29 22:16:00.824914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824895 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e"} err="failed to get container status \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": rpc error: code = NotFound desc = could not find container \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": container with ID starting with 1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e not found: ID does not exist" Oct 29 22:16:00.824959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.824915 5610 scope.go:122] "RemoveContainer" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" Oct 29 22:16:00.825134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825116 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436"} err="failed to get container status \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": rpc error: code = NotFound desc = could not find container \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": container with ID starting with c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436 not found: ID does not exist" Oct 29 22:16:00.825189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825135 5610 scope.go:122] "RemoveContainer" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" Oct 29 22:16:00.825340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825324 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627"} err="failed to get container status \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": rpc error: code = NotFound desc = could not find container \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": container with ID starting with 7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627 not found: ID does not exist" Oct 29 22:16:00.825379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825341 5610 scope.go:122] "RemoveContainer" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" Oct 29 22:16:00.825589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825570 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81"} err="failed to get container status \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": rpc error: code = NotFound desc = could not find container \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": container with ID starting with 06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81 not found: ID does not exist" Oct 29 22:16:00.825637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825589 5610 scope.go:122] "RemoveContainer" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" Oct 29 22:16:00.825806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825789 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995"} err="failed to get container status \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": rpc error: code = NotFound desc = could not find container \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": container with ID starting with f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995 not found: ID does not exist" Oct 29 22:16:00.825875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825809 5610 scope.go:122] "RemoveContainer" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" Oct 29 22:16:00.826011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.825991 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672"} err="failed to get container status \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": rpc error: code = NotFound desc = could not find container \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": container with ID starting with ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672 not found: ID does not exist" Oct 29 22:16:00.826057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826012 5610 scope.go:122] "RemoveContainer" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" Oct 29 22:16:00.826222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826204 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100"} err="failed to get container status \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": rpc error: code = NotFound desc = could not find container \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": container with ID starting with 0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100 not found: ID does not exist" Oct 29 22:16:00.826287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826222 5610 scope.go:122] "RemoveContainer" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" Oct 29 22:16:00.826459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826442 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e"} err="failed to get container status \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": rpc error: code = NotFound desc = could not find container \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": container with ID starting with 1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e not found: ID does not exist" Oct 29 22:16:00.826511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826460 5610 scope.go:122] "RemoveContainer" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" Oct 29 22:16:00.826680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826663 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436"} err="failed to get container status \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": rpc error: code = NotFound desc = could not find container \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": container with ID starting with c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436 not found: ID does not exist" Oct 29 22:16:00.826728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826681 5610 scope.go:122] "RemoveContainer" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" Oct 29 22:16:00.826869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826852 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627"} err="failed to get container status \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": rpc error: code = NotFound desc = could not find container \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": container with ID starting with 7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627 not found: ID does not exist" Oct 29 22:16:00.826908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.826869 5610 scope.go:122] "RemoveContainer" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" Oct 29 22:16:00.827058 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827042 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81"} err="failed to get container status \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": rpc error: code = NotFound desc = could not find container \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": container with ID starting with 06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81 not found: ID does not exist" Oct 29 22:16:00.827100 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827058 5610 scope.go:122] "RemoveContainer" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" Oct 29 22:16:00.827254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827234 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995"} err="failed to get container status \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": rpc error: code = NotFound desc = could not find container \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": container with ID starting with f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995 not found: ID does not exist" Oct 29 22:16:00.827291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827255 5610 scope.go:122] "RemoveContainer" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" Oct 29 22:16:00.827504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827484 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672"} err="failed to get container status \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": rpc error: code = NotFound desc = could not find container \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": container with ID starting with ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672 not found: ID does not exist" Oct 29 22:16:00.827553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827504 5610 scope.go:122] "RemoveContainer" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" Oct 29 22:16:00.827711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827693 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100"} err="failed to get container status \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": rpc error: code = NotFound desc = could not find container \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": container with ID starting with 0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100 not found: ID does not exist" Oct 29 22:16:00.827751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827711 5610 scope.go:122] "RemoveContainer" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" Oct 29 22:16:00.827914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827896 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e"} err="failed to get container status \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": rpc error: code = NotFound desc = could not find container \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": container with ID starting with 1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e not found: ID does not exist" Oct 29 22:16:00.827984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.827916 5610 scope.go:122] "RemoveContainer" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" Oct 29 22:16:00.828155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.828139 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436"} err="failed to get container status \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": rpc error: code = NotFound desc = could not find container \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": container with ID starting with c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436 not found: ID does not exist" Oct 29 22:16:00.828201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.828155 5610 scope.go:122] "RemoveContainer" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" Oct 29 22:16:00.828351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.828332 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627"} err="failed to get container status \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": rpc error: code = NotFound desc = could not find container \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": container with ID starting with 7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627 not found: ID does not exist" Oct 29 22:16:00.828431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.828351 5610 scope.go:122] "RemoveContainer" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" Oct 29 22:16:00.828564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.828550 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81"} err="failed to get container status \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": rpc error: code = NotFound desc = could not find container \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": container with ID starting with 06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81 not found: ID does not exist" Oct 29 22:16:00.828611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.828564 5610 scope.go:122] "RemoveContainer" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" Oct 29 22:16:00.828771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.828753 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995"} err="failed to get container status \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": rpc error: code = NotFound desc = could not find container \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": container with ID starting with f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995 not found: ID does not exist" Oct 29 22:16:00.828840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.828771 5610 scope.go:122] "RemoveContainer" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" Oct 29 22:16:00.829045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829011 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672"} err="failed to get container status \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": rpc error: code = NotFound desc = could not find container \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": container with ID starting with ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672 not found: ID does not exist" Oct 29 22:16:00.829045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829028 5610 scope.go:122] "RemoveContainer" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" Oct 29 22:16:00.829262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829243 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100"} err="failed to get container status \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": rpc error: code = NotFound desc = could not find container \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": container with ID starting with 0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100 not found: ID does not exist" Oct 29 22:16:00.829262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829262 5610 scope.go:122] "RemoveContainer" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" Oct 29 22:16:00.829575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829540 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e"} err="failed to get container status \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": rpc error: code = NotFound desc = could not find container \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": container with ID starting with 1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e not found: ID does not exist" Oct 29 22:16:00.829575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829564 5610 scope.go:122] "RemoveContainer" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" Oct 29 22:16:00.829857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829840 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:16:00.829937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829913 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436"} err="failed to get container status \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": rpc error: code = NotFound desc = could not find container \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": container with ID starting with c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436 not found: ID does not exist" Oct 29 22:16:00.829985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.829939 5610 scope.go:122] "RemoveContainer" containerID="7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627" Oct 29 22:16:00.830262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.830240 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627"} err="failed to get container status \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": rpc error: code = NotFound desc = could not find container \"7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627\": container with ID starting with 7382239b7bb679818896fa0440a9595732e972bad19f0334c8267eabf239c627 not found: ID does not exist" Oct 29 22:16:00.830324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.830263 5610 scope.go:122] "RemoveContainer" containerID="06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81" Oct 29 22:16:00.830522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.830504 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81"} err="failed to get container status \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": rpc error: code = NotFound desc = could not find container \"06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81\": container with ID starting with 06ada553f9fe095c55e4347123a8bd7f774efe05ec59778f2e4936447ade7b81 not found: ID does not exist" Oct 29 22:16:00.830576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.830523 5610 scope.go:122] "RemoveContainer" containerID="f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995" Oct 29 22:16:00.830750 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.830730 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995"} err="failed to get container status \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": rpc error: code = NotFound desc = could not find container \"f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995\": container with ID starting with f4c5638eb9cd62cafec6aacebb2a1fd1bae016fdaa7713c593075889b6fc5995 not found: ID does not exist" Oct 29 22:16:00.830783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.830751 5610 scope.go:122] "RemoveContainer" containerID="ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672" Oct 29 22:16:00.830958 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.830941 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672"} err="failed to get container status \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": rpc error: code = NotFound desc = could not find container \"ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672\": container with ID starting with ab03842db946f258b8a34c46d0b7973f918c400df0a2a66767693bc3a3ac8672 not found: ID does not exist" Oct 29 22:16:00.831005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.830959 5610 scope.go:122] "RemoveContainer" containerID="0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100" Oct 29 22:16:00.831211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.831193 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100"} err="failed to get container status \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": rpc error: code = NotFound desc = could not find container \"0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100\": container with ID starting with 0f7f70431191e31aa359f2bcd8f939eb6bfe7ab05f792d3097baeb38bd4ab100 not found: ID does not exist" Oct 29 22:16:00.831245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.831212 5610 scope.go:122] "RemoveContainer" containerID="1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e" Oct 29 22:16:00.831443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.831423 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e"} err="failed to get container status \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": rpc error: code = NotFound desc = could not find container \"1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e\": container with ID starting with 1c60e0e905ab37070389111135719febc283916bb97a7097302e9754292b0a0e not found: ID does not exist" Oct 29 22:16:00.831495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.831443 5610 scope.go:122] "RemoveContainer" containerID="c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436" Oct 29 22:16:00.831685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.831666 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436"} err="failed to get container status \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": rpc error: code = NotFound desc = could not find container \"c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436\": container with ID starting with c30942c85daab712c7a782c176020571164c40c67f58b7373f3400c13e99c436 not found: ID does not exist" Oct 29 22:16:00.839815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.839794 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:16:00.858871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.858843 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:16:00.859162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859148 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy-web" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859165 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy-web" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859178 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="init-config-reloader" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859183 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="init-config-reloader" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859191 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859196 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859204 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy-thanos" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859209 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy-thanos" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859219 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="thanos-sidecar" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859223 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="thanos-sidecar" Oct 29 22:16:00.859227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859230 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="config-reloader" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859244 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="config-reloader" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859261 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="prometheus" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859265 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="prometheus" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859310 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy-thanos" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859321 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="thanos-sidecar" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859329 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="config-reloader" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859336 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="prometheus" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859343 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy-web" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859349 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="init-config-reloader" Oct 29 22:16:00.859584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.859354 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" containerName="kube-rbac-proxy" Oct 29 22:16:00.863291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.863270 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.876127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876106 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 29 22:16:00.876256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876190 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 29 22:16:00.876332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876231 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 29 22:16:00.876332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876313 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 29 22:16:00.876555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876540 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 29 22:16:00.876770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876757 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 29 22:16:00.876770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876764 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 29 22:16:00.876904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876760 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 29 22:16:00.876904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876796 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 29 22:16:00.876904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876825 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 29 22:16:00.877103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876913 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kptl2\"" Oct 29 22:16:00.877103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.876921 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7kpma9rsabkp1\"" Oct 29 22:16:00.877103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.877008 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 29 22:16:00.877103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.877077 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 29 22:16:00.877245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.877121 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 29 22:16:00.891135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.889973 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:16:00.891695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.891615 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 29 22:16:00.892305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.892279 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 29 22:16:00.940560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940515 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.940560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940558 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940587 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940660 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940705 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a4657df-fb28-427e-bcd1-b4d0ba583369-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940724 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5nk\" (UniqueName: \"kubernetes.io/projected/4a4657df-fb28-427e-bcd1-b4d0ba583369-kube-api-access-kd5nk\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940746 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940783 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940882 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940901 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.940918 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941003 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941027 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941046 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941075 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941136 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-config\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941234 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941253 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941273 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a4657df-fb28-427e-bcd1-b4d0ba583369-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:00.941470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:00.941294 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042001 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042056 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042079 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042105 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-config\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042132 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042153 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042179 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a4657df-fb28-427e-bcd1-b4d0ba583369-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042203 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042234 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042257 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042280 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042301 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042330 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a4657df-fb28-427e-bcd1-b4d0ba583369-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042353 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5nk\" (UniqueName: \"kubernetes.io/projected/4a4657df-fb28-427e-bcd1-b4d0ba583369-kube-api-access-kd5nk\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042382 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.042459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042435 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.043046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042468 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.043046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042500 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.043046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042528 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.043046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042526 5610 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 29 22:16:01.043046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042570 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.043046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042984 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.043046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.042988 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.043424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.043059 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.043424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.043290 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.045019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.044190 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.045019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.044717 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.046093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.045664 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.046093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.046047 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.046242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.046171 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.046542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.046512 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.046747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.046689 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a4657df-fb28-427e-bcd1-b4d0ba583369-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.046747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.046733 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.046958 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.046935 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.047112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.047090 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a4657df-fb28-427e-bcd1-b4d0ba583369-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.047411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.047373 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a4657df-fb28-427e-bcd1-b4d0ba583369-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.047965 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.047936 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-config\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.048113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.048093 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.048242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.048227 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.048952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.048934 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a4657df-fb28-427e-bcd1-b4d0ba583369-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.060954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.060935 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5nk\" (UniqueName: \"kubernetes.io/projected/4a4657df-fb28-427e-bcd1-b4d0ba583369-kube-api-access-kd5nk\") pod \"prometheus-k8s-0\" (UID: \"4a4657df-fb28-427e-bcd1-b4d0ba583369\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.180086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.180051 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:01.499266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.499235 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 29 22:16:01.502356 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:16:01.502327 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4657df_fb28_427e_bcd1_b4d0ba583369.slice/crio-7ac080e237d3fb5e924bc6b1f1507ed6a23e9dab5c7a7d76f332852e26b8eb83 WatchSource:0}: Error finding container 7ac080e237d3fb5e924bc6b1f1507ed6a23e9dab5c7a7d76f332852e26b8eb83: Status 404 returned error can't find the container with id 7ac080e237d3fb5e924bc6b1f1507ed6a23e9dab5c7a7d76f332852e26b8eb83 Oct 29 22:16:01.633530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.633495 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0efed476-7ba2-4cce-9d12-eb2e1e457160" path="/var/lib/kubelet/pods/0efed476-7ba2-4cce-9d12-eb2e1e457160/volumes" Oct 29 22:16:01.779676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.779591 5610 generic.go:356] "Generic (PLEG): container finished" podID="4a4657df-fb28-427e-bcd1-b4d0ba583369" containerID="89ce618ee331a0695b8c96a4bfeb415590b838191ea9018644e0768039953c1f" exitCode=0 Oct 29 22:16:01.779824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.779678 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a4657df-fb28-427e-bcd1-b4d0ba583369","Type":"ContainerDied","Data":"89ce618ee331a0695b8c96a4bfeb415590b838191ea9018644e0768039953c1f"} Oct 29 22:16:01.779824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:01.779742 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a4657df-fb28-427e-bcd1-b4d0ba583369","Type":"ContainerStarted","Data":"7ac080e237d3fb5e924bc6b1f1507ed6a23e9dab5c7a7d76f332852e26b8eb83"} Oct 29 22:16:02.785659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.785621 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:02.789291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.789263 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a4657df-fb28-427e-bcd1-b4d0ba583369","Type":"ContainerStarted","Data":"142550acd2193493e1ed8334bde70339e15f6084d14b04ba9e1006bec0769e92"} Oct 29 22:16:02.789406 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.789375 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a4657df-fb28-427e-bcd1-b4d0ba583369","Type":"ContainerStarted","Data":"6a603e39943e3fb33b819b9edc625f1d91a03dfe8308d0a8bbb33d9c51d1ed3f"} Oct 29 22:16:02.789466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.789438 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a4657df-fb28-427e-bcd1-b4d0ba583369","Type":"ContainerStarted","Data":"9ad01c057c0d52cfd211f253d775a1fcf91312885ec9a7b0b7a763508214c09e"} Oct 29 22:16:02.789502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.789475 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a4657df-fb28-427e-bcd1-b4d0ba583369","Type":"ContainerStarted","Data":"09b6b4c3a92a15c2d4861add8182ba244acafb5db126f54ba1ca7f45fff2871d"} Oct 29 22:16:02.789532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.789517 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a4657df-fb28-427e-bcd1-b4d0ba583369","Type":"ContainerStarted","Data":"c29ccf1daaf2b6eed8db220660bc47469b6668cef8451b85a21e6873fd9ca459"} Oct 29 22:16:02.789562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.789552 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a4657df-fb28-427e-bcd1-b4d0ba583369","Type":"ContainerStarted","Data":"6b1860fab02354d34b1bb1dd8d21df8d40e172f8cfd9aac0cf92a0d79dbb0056"} Oct 29 22:16:02.842680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.842621 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8266936449999998 podStartE2EDuration="2.842608532s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.914887ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:02.841877466 +0000 UTC m=+25.323194995" watchObservedRunningTime="2026-10-29 22:16:02.842608532 +0000 UTC m=+25.323926022" Oct 29 22:16:02.918244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:02.918206 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:03.084564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:03.084483 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:03.216636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:03.216607 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:06.180927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:06.180883 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:07.588838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:07.588790 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:11.024341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:11.024308 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qq6r4" Oct 29 22:16:13.708236 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:13.708195 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:16:34.440326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:34.440288 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:16:43.708519 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:16:43.708476 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:16:51.062572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:51.062520 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:51.078239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:51.078215 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:51.196563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:51.196534 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 29 22:16:59.912331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:16:59.912295 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:00.048717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:00.048680 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:12.649342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:12.649298 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:12.777868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:12.777833 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:13.707377 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:17:13.707337 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:17:20.836389 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:20.836306 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:20.969358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:20.969320 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:21.141973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:21.141884 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:21.277839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:21.277799 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:29.940271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:29.940234 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:30.071975 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:30.071946 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:32.190144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:32.190102 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:32.349729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:32.349691 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:34.557450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:17:34.557419 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:17:43.707797 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:17:43.707760 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:18:05.377652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:18:05.377621 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:05.509322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:18:05.509271 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:13.708071 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:18:13.708032 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:18:30.583171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:18:30.583132 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:34.679479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:18:34.679442 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:18:43.707475 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:18:43.707428 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:19:07.378036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:07.377989 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:08.983605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:08.983555 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:13.707808 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:19:13.707771 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:19:19.930503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:19.930471 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:20.092314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:20.092283 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:32.533231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:32.533201 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:34.803835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:34.803802 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:35.482189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:35.482156 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:41.586894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:41.586844 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:42.388316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:42.388277 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:43.708126 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:19:43.708082 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:19:44.798231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:44.798198 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:44.963350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:44.963319 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:46.693248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:46.693214 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:46.869951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:46.869918 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:47.026268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:47.026190 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:49.537333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:49.537300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:49.759980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:49.759928 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:50.558775 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:50.558734 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:51.621216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:51.621185 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:51.756149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:51.756117 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:56.567528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:56.567501 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:57.993518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:57.993481 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:19:58.119859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:19:58.119826 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:04.343377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:04.343344 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:04.484283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:04.484231 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:06.608060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:06.608024 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:13.708852 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:20:13.708814 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:20:16.799375 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:16.799344 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:16.960307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:16.960268 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:17.339846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.339809 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:17.478638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.478605 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m"] Oct 29 22:20:17.481168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.481149 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.512288 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.512251 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m"] Oct 29 22:20:17.577850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.577815 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 29 22:20:17.577850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.577858 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 29 22:20:17.578067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.577859 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 29 22:20:17.578126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.578069 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 29 22:20:17.578200 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.578183 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kv9sm\"" Oct 29 22:20:17.578251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.578192 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 29 22:20:17.606028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.605959 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.606028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.606001 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.606028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.606027 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.606264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.606133 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.606264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.606166 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzztn\" (UniqueName: \"kubernetes.io/projected/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-kube-api-access-qzztn\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.606264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.606200 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.636830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.636799 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:17.707660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.707631 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.707805 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.707669 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.707805 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.707690 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzztn\" (UniqueName: \"kubernetes.io/projected/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-kube-api-access-qzztn\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.707805 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.707719 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.707805 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.707752 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.707805 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.707800 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.708244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.708215 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-manifests\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.710077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.710046 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-tmp\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.710187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.710096 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-webhook-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.710241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.710180 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-metrics-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.710278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.710246 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.725753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.725731 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzztn\" (UniqueName: \"kubernetes.io/projected/9b8cb14f-a7e7-438d-bd86-de6e21363fb4-kube-api-access-qzztn\") pod \"opendatahub-operator-controller-manager-b95c7d58c-vzk8m\" (UID: \"9b8cb14f-a7e7-438d-bd86-de6e21363fb4\") " pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.790223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.790192 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:17.794148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.794131 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:17.924772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:17.924688 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:18.130409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:18.130356 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m"] Oct 29 22:20:18.144534 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:20:18.144507 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8cb14f_a7e7_438d_bd86_de6e21363fb4.slice/crio-18913642876b03b937bd9e73da36abd9ccbbcd80cff3dd86d902f6e16c575a9c WatchSource:0}: Error finding container 18913642876b03b937bd9e73da36abd9ccbbcd80cff3dd86d902f6e16c575a9c: Status 404 returned error can't find the container with id 18913642876b03b937bd9e73da36abd9ccbbcd80cff3dd86d902f6e16c575a9c Oct 29 22:20:18.146088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:18.146070 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:20:18.307225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:18.307185 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:18.628276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:18.628194 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" event={"ID":"9b8cb14f-a7e7-438d-bd86-de6e21363fb4","Type":"ContainerStarted","Data":"18913642876b03b937bd9e73da36abd9ccbbcd80cff3dd86d902f6e16c575a9c"} Oct 29 22:20:21.639184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:21.639102 5610 generic.go:356] "Generic (PLEG): container finished" podID="9b8cb14f-a7e7-438d-bd86-de6e21363fb4" containerID="9ed560a7368285381a2e2767492d48f33e39e9d0a96f7a3bbaed1981ba44074e" exitCode=0 Oct 29 22:20:21.639184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:21.639154 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" event={"ID":"9b8cb14f-a7e7-438d-bd86-de6e21363fb4","Type":"ContainerDied","Data":"9ed560a7368285381a2e2767492d48f33e39e9d0a96f7a3bbaed1981ba44074e"} Oct 29 22:20:22.644984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:22.644949 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" event={"ID":"9b8cb14f-a7e7-438d-bd86-de6e21363fb4","Type":"ContainerStarted","Data":"dbb862d493830bf57930f29f462dd08b1b327ff9d59cd8913b6092ebd09e6342"} Oct 29 22:20:22.645362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:22.645089 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:22.712763 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:22.712719 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" podStartSLOduration=2.2448177400000002 podStartE2EDuration="5.712706518s" totalImagesPullingTime="3.017055965s" totalInitContainerRuntime="450.832813ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:22.702040388 +0000 UTC m=+285.183357881" watchObservedRunningTime="2026-10-29 22:20:22.712706518 +0000 UTC m=+285.194024001" Oct 29 22:20:25.473007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:25.472978 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:25.604944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:25.604902 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:31.051924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:31.051885 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:31.203460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:31.203420 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:33.650512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:33.650483 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b95c7d58c-vzk8m" Oct 29 22:20:34.922926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:34.922893 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:36.858412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:36.858356 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:37.013170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.013131 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:37.325932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.325898 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b649d648f-h99f8"] Oct 29 22:20:37.333171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.333144 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.360935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.360898 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 29 22:20:37.361067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.360933 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 29 22:20:37.361067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.360945 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 29 22:20:37.361246 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.361225 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 29 22:20:37.361368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.361288 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5k5vk\"" Oct 29 22:20:37.361737 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.361713 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b649d648f-h99f8"] Oct 29 22:20:37.372254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.372219 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:37.466475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.466449 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-tls-certs\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.466636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.466481 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptq9\" (UniqueName: \"kubernetes.io/projected/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-kube-api-access-4ptq9\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.466636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.466502 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-tmp\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.512962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.512936 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:37.567266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.567226 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-tls-certs\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.567266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.567270 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptq9\" (UniqueName: \"kubernetes.io/projected/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-kube-api-access-4ptq9\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.567496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.567288 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-tmp\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.569565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.569538 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-tmp\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.569753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.569737 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-tls-certs\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.584063 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.584007 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptq9\" (UniqueName: \"kubernetes.io/projected/4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb-kube-api-access-4ptq9\") pod \"kube-auth-proxy-b649d648f-h99f8\" (UID: \"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb\") " pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.655241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.655214 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5k5vk\"" Oct 29 22:20:37.663613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.663591 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" Oct 29 22:20:37.977539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:37.977509 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b649d648f-h99f8"] Oct 29 22:20:37.986266 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:20:37.986234 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e22eb3f_2e6a_46e1_9873_fa5f386cc6fb.slice/crio-665678c12f389628a50dc08cf1d57e2de05d21785f2e99247c775b3016c255a7 WatchSource:0}: Error finding container 665678c12f389628a50dc08cf1d57e2de05d21785f2e99247c775b3016c255a7: Status 404 returned error can't find the container with id 665678c12f389628a50dc08cf1d57e2de05d21785f2e99247c775b3016c255a7 Oct 29 22:20:38.487019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.486978 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz"] Oct 29 22:20:38.489812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.489739 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.505538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.505455 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 29 22:20:38.505757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.505730 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 29 22:20:38.505916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.505898 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 29 22:20:38.505986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.505962 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 29 22:20:38.513070 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.513049 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cvlmz\"" Oct 29 22:20:38.518883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.518856 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz"] Oct 29 22:20:38.575377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.575301 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.575377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.575350 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.575607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.575491 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.575607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.575539 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.575607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.575601 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s4pg\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-kube-api-access-6s4pg\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.575774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.575652 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f96d0e-e4f9-4201-abc3-d473e483850d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.575830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.575784 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.597887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.597849 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:38.676927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.676879 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.677107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.676944 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.677107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.676979 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.677107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.677012 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.677107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.677060 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s4pg\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-kube-api-access-6s4pg\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.677107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.677088 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f96d0e-e4f9-4201-abc3-d473e483850d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.677372 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.677112 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.678447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.678417 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.680272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.680246 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f96d0e-e4f9-4201-abc3-d473e483850d-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.680614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.680586 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.680799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.680779 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.680908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.680883 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.695684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.695659 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s4pg\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-kube-api-access-6s4pg\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.700048 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.700021 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4ldkz\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:38.701212 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.701173 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" event={"ID":"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb","Type":"ContainerStarted","Data":"665678c12f389628a50dc08cf1d57e2de05d21785f2e99247c775b3016c255a7"} Oct 29 22:20:38.723430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.723368 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:38.803494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:38.803387 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:39.121564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:39.121524 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz"] Oct 29 22:20:39.124558 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:20:39.124528 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f96d0e_e4f9_4201_abc3_d473e483850d.slice/crio-d6cc911401b7a3bc82e03a3992224c20bb3ddf3b6290e5b74a3355abd9c447a3 WatchSource:0}: Error finding container d6cc911401b7a3bc82e03a3992224c20bb3ddf3b6290e5b74a3355abd9c447a3: Status 404 returned error can't find the container with id d6cc911401b7a3bc82e03a3992224c20bb3ddf3b6290e5b74a3355abd9c447a3 Oct 29 22:20:39.489505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:39.489462 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:39.707255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:39.707209 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" event={"ID":"b7f96d0e-e4f9-4201-abc3-d473e483850d","Type":"ContainerStarted","Data":"d6cc911401b7a3bc82e03a3992224c20bb3ddf3b6290e5b74a3355abd9c447a3"} Oct 29 22:20:40.330853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.330812 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:40.399483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.399455 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 29 22:20:40.539371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.539336 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-nk5rv"] Oct 29 22:20:40.542090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.542068 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:40.554863 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.554845 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 29 22:20:40.565296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.565243 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-4kn96\"" Oct 29 22:20:40.565989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.565969 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-nk5rv"] Oct 29 22:20:40.639861 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.639734 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:40.697355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.697324 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kpqs\" (UniqueName: \"kubernetes.io/projected/9de87ff9-d183-4132-b1b4-2542a370c998-kube-api-access-6kpqs\") pod \"odh-observability-584f99c788-nk5rv\" (UID: \"9de87ff9-d183-4132-b1b4-2542a370c998\") " pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:40.697552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.697486 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/9de87ff9-d183-4132-b1b4-2542a370c998-webhook-certs\") pod \"odh-observability-584f99c788-nk5rv\" (UID: \"9de87ff9-d183-4132-b1b4-2542a370c998\") " pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:40.713296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.713268 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" event={"ID":"4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb","Type":"ContainerStarted","Data":"5ed5231792d41faddbb10c68792df08c55e207f08f810cd7a15368d44d71d9f2"} Oct 29 22:20:40.748629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.748585 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b649d648f-h99f8" podStartSLOduration=1.349609375 podStartE2EDuration="3.748572282s" totalImagesPullingTime="2.398962907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:40.748427304 +0000 UTC m=+303.229744801" watchObservedRunningTime="2026-10-29 22:20:40.748572282 +0000 UTC m=+303.229889818" Oct 29 22:20:40.798748 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.798713 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpqs\" (UniqueName: \"kubernetes.io/projected/9de87ff9-d183-4132-b1b4-2542a370c998-kube-api-access-6kpqs\") pod \"odh-observability-584f99c788-nk5rv\" (UID: \"9de87ff9-d183-4132-b1b4-2542a370c998\") " pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:40.798932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.798837 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/9de87ff9-d183-4132-b1b4-2542a370c998-webhook-certs\") pod \"odh-observability-584f99c788-nk5rv\" (UID: \"9de87ff9-d183-4132-b1b4-2542a370c998\") " pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:40.799018 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:20:40.798998 5610 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 29 22:20:40.799079 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:20:40.799070 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de87ff9-d183-4132-b1b4-2542a370c998-webhook-certs podName:9de87ff9-d183-4132-b1b4-2542a370c998 nodeName:}" failed. No retries permitted until 2026-10-29 22:20:41.299049026 +0000 UTC m=+303.780366502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/9de87ff9-d183-4132-b1b4-2542a370c998-webhook-certs") pod "odh-observability-584f99c788-nk5rv" (UID: "9de87ff9-d183-4132-b1b4-2542a370c998") : secret "odh-observability-webhook-cert" not found Oct 29 22:20:40.822170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:40.822137 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpqs\" (UniqueName: \"kubernetes.io/projected/9de87ff9-d183-4132-b1b4-2542a370c998-kube-api-access-6kpqs\") pod \"odh-observability-584f99c788-nk5rv\" (UID: \"9de87ff9-d183-4132-b1b4-2542a370c998\") " pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:41.046970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:41.046916 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:41.303589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:41.303499 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/9de87ff9-d183-4132-b1b4-2542a370c998-webhook-certs\") pod \"odh-observability-584f99c788-nk5rv\" (UID: \"9de87ff9-d183-4132-b1b4-2542a370c998\") " pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:41.306121 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:41.306093 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/9de87ff9-d183-4132-b1b4-2542a370c998-webhook-certs\") pod \"odh-observability-584f99c788-nk5rv\" (UID: \"9de87ff9-d183-4132-b1b4-2542a370c998\") " pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:41.455082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:41.455044 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:41.664946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:41.664250 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:20:41.664946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:41.664332 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:20:41.718738 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:41.718707 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" event={"ID":"b7f96d0e-e4f9-4201-abc3-d473e483850d","Type":"ContainerStarted","Data":"b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860"} Oct 29 22:20:41.754326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:41.754290 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-nk5rv"] Oct 29 22:20:41.761329 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:20:41.761305 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de87ff9_d183_4132_b1b4_2542a370c998.slice/crio-9f3b58c32f1853f6d13103d8966c6a0a501b111aa0c7b7da3cecec83c9e9c49b WatchSource:0}: Error finding container 9f3b58c32f1853f6d13103d8966c6a0a501b111aa0c7b7da3cecec83c9e9c49b: Status 404 returned error can't find the container with id 9f3b58c32f1853f6d13103d8966c6a0a501b111aa0c7b7da3cecec83c9e9c49b Oct 29 22:20:42.486780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:42.486739 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:42.616578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:42.616540 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:42.725041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:42.725000 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-nk5rv" event={"ID":"9de87ff9-d183-4132-b1b4-2542a370c998","Type":"ContainerStarted","Data":"9f3b58c32f1853f6d13103d8966c6a0a501b111aa0c7b7da3cecec83c9e9c49b"} Oct 29 22:20:42.801566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:42.801456 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" podStartSLOduration=2.264827734 podStartE2EDuration="4.801439646s" totalImagesPullingTime="2.536611912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:42.790632289 +0000 UTC m=+305.271949785" watchObservedRunningTime="2026-10-29 22:20:42.801439646 +0000 UTC m=+305.282757140" Oct 29 22:20:43.113336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:43.113258 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:43.242718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:43.242679 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:43.460047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:43.460015 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:43.714601 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:20:43.714524 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:20:43.729803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:43.729772 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:43.731862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:43.731655 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:20:44.734440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:44.734386 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-nk5rv" event={"ID":"9de87ff9-d183-4132-b1b4-2542a370c998","Type":"ContainerStarted","Data":"a18621cbe78af378d55da9ab1f21456715369a48c35347d788f8bf7014edfe9c"} Oct 29 22:20:44.775890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:44.775836 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-nk5rv" podStartSLOduration=2.469006619 podStartE2EDuration="4.775824349s" totalImagesPullingTime="2.30681773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:20:44.774924252 +0000 UTC m=+307.256241744" watchObservedRunningTime="2026-10-29 22:20:44.775824349 +0000 UTC m=+307.257141842" Oct 29 22:20:45.738211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:45.738178 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:20:53.098410 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:53.098368 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:53.309335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:53.309301 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:20:56.742982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:20:56.742942 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-nk5rv" Oct 29 22:21:09.473380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:09.473347 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:09.606578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:09.606548 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:13.709063 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:21:13.709028 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:21:19.098380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:19.098344 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:35.086857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:35.086820 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:38.382849 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.382814 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds"] Oct 29 22:21:38.385135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.385109 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.412036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.412008 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds"] Oct 29 22:21:38.496308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.496277 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31e7e56-219b-4c05-8868-77037656ae70-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.496308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.496312 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.496537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.496336 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.496537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.496417 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31e7e56-219b-4c05-8868-77037656ae70-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.496537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.496493 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d31e7e56-219b-4c05-8868-77037656ae70-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.496638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.496563 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqgbq\" (UniqueName: \"kubernetes.io/projected/d31e7e56-219b-4c05-8868-77037656ae70-kube-api-access-vqgbq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.496638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.496592 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.597776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.597746 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31e7e56-219b-4c05-8868-77037656ae70-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.597776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.597781 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.598015 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.597803 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.598015 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.597831 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31e7e56-219b-4c05-8868-77037656ae70-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.598015 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.597859 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d31e7e56-219b-4c05-8868-77037656ae70-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.598015 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.597899 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqgbq\" (UniqueName: \"kubernetes.io/projected/d31e7e56-219b-4c05-8868-77037656ae70-kube-api-access-vqgbq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.598015 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.597932 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.598654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.598623 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d31e7e56-219b-4c05-8868-77037656ae70-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.600237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.600207 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.600349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.600257 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.600470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.600452 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31e7e56-219b-4c05-8868-77037656ae70-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.600661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.600638 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d31e7e56-219b-4c05-8868-77037656ae70-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.618717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.618695 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqgbq\" (UniqueName: \"kubernetes.io/projected/d31e7e56-219b-4c05-8868-77037656ae70-kube-api-access-vqgbq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.618817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.618731 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31e7e56-219b-4c05-8868-77037656ae70-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kg9ds\" (UID: \"d31e7e56-219b-4c05-8868-77037656ae70\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.695855 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.695830 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:38.999192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:38.999166 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds"] Oct 29 22:21:39.001486 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:21:39.001462 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31e7e56_219b_4c05_8868_77037656ae70.slice/crio-001640fe939cc802a9cab3e57c7da529c13696432914fa5c2d2f1f6603274b2c WatchSource:0}: Error finding container 001640fe939cc802a9cab3e57c7da529c13696432914fa5c2d2f1f6603274b2c: Status 404 returned error can't find the container with id 001640fe939cc802a9cab3e57c7da529c13696432914fa5c2d2f1f6603274b2c Oct 29 22:21:39.003642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:39.003612 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:21:39.003713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:39.003678 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:21:39.725774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:39.725735 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:39.867265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:39.867225 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:39.937247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:39.937216 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" event={"ID":"d31e7e56-219b-4c05-8868-77037656ae70","Type":"ContainerStarted","Data":"af102cffc91bd82d132262d1c6a565b1a97f96e439fa106dd6de4a308012e9ac"} Oct 29 22:21:39.937430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:39.937269 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" event={"ID":"d31e7e56-219b-4c05-8868-77037656ae70","Type":"ContainerStarted","Data":"001640fe939cc802a9cab3e57c7da529c13696432914fa5c2d2f1f6603274b2c"} Oct 29 22:21:39.937430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:39.937361 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:40.006516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:40.006376 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" podStartSLOduration=2.006363772 podStartE2EDuration="2.006363772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:21:39.995833689 +0000 UTC m=+362.477151182" watchObservedRunningTime="2026-10-29 22:21:40.006363772 +0000 UTC m=+362.487681265" Oct 29 22:21:40.057518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:40.057482 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:40.194653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:40.194617 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:40.539944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:40.539910 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:40.942420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:40.942337 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kg9ds" Oct 29 22:21:41.018814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.018781 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz"] Oct 29 22:21:41.019054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.019031 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" podUID="b7f96d0e-e4f9-4201-abc3-d473e483850d" containerName="discovery" containerID="cri-o://b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860" gracePeriod=30 Oct 29 22:21:41.260143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.260120 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:21:41.320828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.320793 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-kube-api-access-6s4pg\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-kube-api-access-6s4pg\") pod \"b7f96d0e-e4f9-4201-abc3-d473e483850d\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " Oct 29 22:21:41.321004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.320839 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-cacerts\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-cacerts\") pod \"b7f96d0e-e4f9-4201-abc3-d473e483850d\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " Oct 29 22:21:41.321004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.320871 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f96d0e-e4f9-4201-abc3-d473e483850d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f96d0e-e4f9-4201-abc3-d473e483850d-local-certs\") pod \"b7f96d0e-e4f9-4201-abc3-d473e483850d\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " Oct 29 22:21:41.321004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.320921 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-token\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-token\") pod \"b7f96d0e-e4f9-4201-abc3-d473e483850d\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " Oct 29 22:21:41.321169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.321062 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-dns-cert\") pod \"b7f96d0e-e4f9-4201-abc3-d473e483850d\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " Oct 29 22:21:41.321169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.321129 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-ca-configmap\") pod \"b7f96d0e-e4f9-4201-abc3-d473e483850d\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " Oct 29 22:21:41.321285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.321180 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-kubeconfig\") pod \"b7f96d0e-e4f9-4201-abc3-d473e483850d\" (UID: \"b7f96d0e-e4f9-4201-abc3-d473e483850d\") " Oct 29 22:21:41.321795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.321723 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-ca-configmap" pod "b7f96d0e-e4f9-4201-abc3-d473e483850d" (UID: "b7f96d0e-e4f9-4201-abc3-d473e483850d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:21:41.323335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.323297 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-token" pod "b7f96d0e-e4f9-4201-abc3-d473e483850d" (UID: "b7f96d0e-e4f9-4201-abc3-d473e483850d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:41.323462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.323441 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-kubeconfig" pod "b7f96d0e-e4f9-4201-abc3-d473e483850d" (UID: "b7f96d0e-e4f9-4201-abc3-d473e483850d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:21:41.323569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.323541 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-cacerts" pod "b7f96d0e-e4f9-4201-abc3-d473e483850d" (UID: "b7f96d0e-e4f9-4201-abc3-d473e483850d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:21:41.323691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.323604 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-kube-api-access-6s4pg" pod "b7f96d0e-e4f9-4201-abc3-d473e483850d" (UID: "b7f96d0e-e4f9-4201-abc3-d473e483850d"). InnerVolumeSpecName "kube-api-access-6s4pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:21:41.323691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.323608 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f96d0e-e4f9-4201-abc3-d473e483850d-local-certs" pod "b7f96d0e-e4f9-4201-abc3-d473e483850d" (UID: "b7f96d0e-e4f9-4201-abc3-d473e483850d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:21:41.323691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.323612 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-dns-cert" pod "b7f96d0e-e4f9-4201-abc3-d473e483850d" (UID: "b7f96d0e-e4f9-4201-abc3-d473e483850d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:21:41.422127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.422094 5610 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-ca-configmap\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:21:41.422127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.422123 5610 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-kubeconfig\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:21:41.422127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.422133 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s4pg\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-kube-api-access-6s4pg\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:21:41.422338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.422143 5610 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-cacerts\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:21:41.422338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.422153 5610 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f96d0e-e4f9-4201-abc3-d473e483850d-local-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:21:41.422338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.422161 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:21:41.422338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.422169 5610 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b7f96d0e-e4f9-4201-abc3-d473e483850d-istio-csr-dns-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:21:41.945533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.945505 5610 generic.go:356] "Generic (PLEG): container finished" podID="b7f96d0e-e4f9-4201-abc3-d473e483850d" containerID="b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860" exitCode=0 Oct 29 22:21:41.945918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.945558 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" Oct 29 22:21:41.945918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.945592 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" event={"ID":"b7f96d0e-e4f9-4201-abc3-d473e483850d","Type":"ContainerDied","Data":"b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860"} Oct 29 22:21:41.945918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.945647 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz" event={"ID":"b7f96d0e-e4f9-4201-abc3-d473e483850d","Type":"ContainerDied","Data":"d6cc911401b7a3bc82e03a3992224c20bb3ddf3b6290e5b74a3355abd9c447a3"} Oct 29 22:21:41.945918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.945669 5610 scope.go:122] "RemoveContainer" containerID="b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860" Oct 29 22:21:41.953521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.953507 5610 scope.go:122] "RemoveContainer" containerID="b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860" Oct 29 22:21:41.953766 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:21:41.953749 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860\": container with ID starting with b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860 not found: ID does not exist" containerID="b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860" Oct 29 22:21:41.953803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.953775 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860"} err="failed to get container status \"b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860\": rpc error: code = NotFound desc = could not find container \"b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860\": container with ID starting with b255c99adb231f7351a5b7c5005918a4f1405afcf148c71e7c62b3d161e5b860 not found: ID does not exist" Oct 29 22:21:41.999881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:41.999847 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz"] Oct 29 22:21:42.010113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:42.010091 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4ldkz"] Oct 29 22:21:42.433786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:42.433702 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:42.571626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:42.571595 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:42.698601 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:42.698573 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:42.831130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:42.831097 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:42.990311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:42.990234 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:21:43.633419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:21:43.633377 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f96d0e-e4f9-4201-abc3-d473e483850d" path="/var/lib/kubelet/pods/b7f96d0e-e4f9-4201-abc3-d473e483850d/volumes" Oct 29 22:21:43.708747 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:21:43.708712 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:22:13.708204 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:22:13.708168 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:22:30.122876 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:30.122843 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:30.273002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:30.272972 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:35.158421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:35.158354 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:43.708778 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:22:43.708740 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:22:46.271229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:46.271195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:48.247388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.247355 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw"] Oct 29 22:22:48.247811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.247688 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b7f96d0e-e4f9-4201-abc3-d473e483850d" containerName="discovery" Oct 29 22:22:48.247811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.247700 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f96d0e-e4f9-4201-abc3-d473e483850d" containerName="discovery" Oct 29 22:22:48.247811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.247761 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f96d0e-e4f9-4201-abc3-d473e483850d" containerName="discovery" Oct 29 22:22:48.252020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.252000 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:22:48.274669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.274640 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qff74\"" Oct 29 22:22:48.274669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.274643 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 29 22:22:48.274825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.274643 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 29 22:22:48.275818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.275799 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw"] Oct 29 22:22:48.285548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.285516 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxwxm\" (UniqueName: \"kubernetes.io/projected/21bc058d-2cfa-4033-ba92-45ddfe6999f3-kube-api-access-cxwxm\") pod \"limitador-operator-controller-manager-856dd584c5-rt2bw\" (UID: \"21bc058d-2cfa-4033-ba92-45ddfe6999f3\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:22:48.387033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.386998 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxwxm\" (UniqueName: \"kubernetes.io/projected/21bc058d-2cfa-4033-ba92-45ddfe6999f3-kube-api-access-cxwxm\") pod \"limitador-operator-controller-manager-856dd584c5-rt2bw\" (UID: \"21bc058d-2cfa-4033-ba92-45ddfe6999f3\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:22:48.411298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.411248 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxwxm\" (UniqueName: \"kubernetes.io/projected/21bc058d-2cfa-4033-ba92-45ddfe6999f3-kube-api-access-cxwxm\") pod \"limitador-operator-controller-manager-856dd584c5-rt2bw\" (UID: \"21bc058d-2cfa-4033-ba92-45ddfe6999f3\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:22:48.564646 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.564559 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:22:48.857520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.857431 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw"] Oct 29 22:22:48.860639 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:22:48.860613 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bc058d_2cfa_4033_ba92_45ddfe6999f3.slice/crio-59b751d59000de334b88bd487498511b8bf029c7023c3dd21d8bbfaac57e2f5d WatchSource:0}: Error finding container 59b751d59000de334b88bd487498511b8bf029c7023c3dd21d8bbfaac57e2f5d: Status 404 returned error can't find the container with id 59b751d59000de334b88bd487498511b8bf029c7023c3dd21d8bbfaac57e2f5d Oct 29 22:22:48.882801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:48.882706 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:49.194662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:49.194627 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" event={"ID":"21bc058d-2cfa-4033-ba92-45ddfe6999f3","Type":"ContainerStarted","Data":"59b751d59000de334b88bd487498511b8bf029c7023c3dd21d8bbfaac57e2f5d"} Oct 29 22:22:49.268165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:49.268122 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:49.680059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:49.679973 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:52.208555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.208520 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" event={"ID":"21bc058d-2cfa-4033-ba92-45ddfe6999f3","Type":"ContainerStarted","Data":"4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d"} Oct 29 22:22:52.209058 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.208756 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:22:52.223849 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.223821 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd"] Oct 29 22:22:52.227246 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.227228 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:52.241329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.241311 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qjqmw\"" Oct 29 22:22:52.245627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.245601 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd"] Oct 29 22:22:52.256374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.256336 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" podStartSLOduration=1.8730213930000001 podStartE2EDuration="4.256326032s" totalImagesPullingTime="2.383304639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:52.255777234 +0000 UTC m=+434.737094727" watchObservedRunningTime="2026-10-29 22:22:52.256326032 +0000 UTC m=+434.737643524" Oct 29 22:22:52.328411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.328368 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svqtl\" (UniqueName: \"kubernetes.io/projected/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-kube-api-access-svqtl\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" (UID: \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:52.328580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.328510 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" (UID: \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:52.429898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.429865 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" (UID: \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:52.430222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.429933 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svqtl\" (UniqueName: \"kubernetes.io/projected/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-kube-api-access-svqtl\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" (UID: \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:52.430288 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.430253 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" (UID: \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:52.448000 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.447975 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svqtl\" (UniqueName: \"kubernetes.io/projected/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-kube-api-access-svqtl\") pod \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" (UID: \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:52.463449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.463371 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:52.540102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.540063 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:52.725224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.725141 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:52.854099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.854056 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd"] Oct 29 22:22:52.857103 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:22:52.857078 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b76c04f_c8b7_4917_8a0f_5dca81eabb31.slice/crio-463a528ddb4b974c166af8966ded1b9cfc08250c35d733d336454675aa1a4252 WatchSource:0}: Error finding container 463a528ddb4b974c166af8966ded1b9cfc08250c35d733d336454675aa1a4252: Status 404 returned error can't find the container with id 463a528ddb4b974c166af8966ded1b9cfc08250c35d733d336454675aa1a4252 Oct 29 22:22:52.883406 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:52.883373 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:53.213884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:53.213848 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" event={"ID":"1b76c04f-c8b7-4917-8a0f-5dca81eabb31","Type":"ContainerStarted","Data":"463a528ddb4b974c166af8966ded1b9cfc08250c35d733d336454675aa1a4252"} Oct 29 22:22:53.752831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:53.752781 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:53.913956 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:53.913920 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:54.800830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:54.800800 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-9g5gh"] Oct 29 22:22:54.804587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:54.804562 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" Oct 29 22:22:54.828386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:54.828362 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-9g5gh"] Oct 29 22:22:54.850008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:54.849969 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd58p\" (UniqueName: \"kubernetes.io/projected/8154e098-0d55-4597-81fe-74d4dae02b3d-kube-api-access-qd58p\") pod \"authorino-operator-68d6dd76dc-9g5gh\" (UID: \"8154e098-0d55-4597-81fe-74d4dae02b3d\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" Oct 29 22:22:54.884217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:54.884188 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2j2gs\"" Oct 29 22:22:54.951504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:54.951432 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd58p\" (UniqueName: \"kubernetes.io/projected/8154e098-0d55-4597-81fe-74d4dae02b3d-kube-api-access-qd58p\") pod \"authorino-operator-68d6dd76dc-9g5gh\" (UID: \"8154e098-0d55-4597-81fe-74d4dae02b3d\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" Oct 29 22:22:54.972412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:54.972355 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd58p\" (UniqueName: \"kubernetes.io/projected/8154e098-0d55-4597-81fe-74d4dae02b3d-kube-api-access-qd58p\") pod \"authorino-operator-68d6dd76dc-9g5gh\" (UID: \"8154e098-0d55-4597-81fe-74d4dae02b3d\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" Oct 29 22:22:55.121043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:55.120961 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" Oct 29 22:22:55.329145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:55.329110 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:55.424232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:55.424210 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-9g5gh"] Oct 29 22:22:55.427261 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:22:55.427227 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8154e098_0d55_4597_81fe_74d4dae02b3d.slice/crio-5a65de571c1b51edc020a4dc2d865884ee5cf549757b2d4f01abb78edcc7a6bc WatchSource:0}: Error finding container 5a65de571c1b51edc020a4dc2d865884ee5cf549757b2d4f01abb78edcc7a6bc: Status 404 returned error can't find the container with id 5a65de571c1b51edc020a4dc2d865884ee5cf549757b2d4f01abb78edcc7a6bc Oct 29 22:22:55.476966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:55.476935 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:56.230113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:56.230073 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" event={"ID":"8154e098-0d55-4597-81fe-74d4dae02b3d","Type":"ContainerStarted","Data":"5a65de571c1b51edc020a4dc2d865884ee5cf549757b2d4f01abb78edcc7a6bc"} Oct 29 22:22:56.716656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:56.716624 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:57.018213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.018180 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js"] Oct 29 22:22:57.021809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.021789 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" Oct 29 22:22:57.051345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.051327 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 29 22:22:57.051345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.051329 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tlm8b\"" Oct 29 22:22:57.054772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.054751 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js"] Oct 29 22:22:57.071526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.071500 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwkj\" (UniqueName: \"kubernetes.io/projected/1f14e09a-672a-4d00-b240-c940f41939a6-kube-api-access-smwkj\") pod \"dns-operator-controller-manager-65bfd79598-6p7js\" (UID: \"1f14e09a-672a-4d00-b240-c940f41939a6\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" Oct 29 22:22:57.172744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.172712 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwkj\" (UniqueName: \"kubernetes.io/projected/1f14e09a-672a-4d00-b240-c940f41939a6-kube-api-access-smwkj\") pod \"dns-operator-controller-manager-65bfd79598-6p7js\" (UID: \"1f14e09a-672a-4d00-b240-c940f41939a6\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" Oct 29 22:22:57.190157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.190127 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwkj\" (UniqueName: \"kubernetes.io/projected/1f14e09a-672a-4d00-b240-c940f41939a6-kube-api-access-smwkj\") pod \"dns-operator-controller-manager-65bfd79598-6p7js\" (UID: \"1f14e09a-672a-4d00-b240-c940f41939a6\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" Oct 29 22:22:57.236353 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.236327 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" event={"ID":"8154e098-0d55-4597-81fe-74d4dae02b3d","Type":"ContainerStarted","Data":"de181ddf801aa4870f15a8429edd9185c6a24afbaa3416e13b0ac0df2df4e563"} Oct 29 22:22:57.236808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.236442 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" Oct 29 22:22:57.238156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.238128 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" event={"ID":"1b76c04f-c8b7-4917-8a0f-5dca81eabb31","Type":"ContainerStarted","Data":"79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091"} Oct 29 22:22:57.238255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.238183 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:22:57.272348 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.272266 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" podStartSLOduration=2.115812693 podStartE2EDuration="3.272253491s" totalImagesPullingTime="1.156440798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:57.271616111 +0000 UTC m=+439.752933607" watchObservedRunningTime="2026-10-29 22:22:57.272253491 +0000 UTC m=+439.753570984" Oct 29 22:22:57.310306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.310253 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" podStartSLOduration=1.764603123 podStartE2EDuration="5.310237648s" totalImagesPullingTime="3.545634525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:57.309815133 +0000 UTC m=+439.791132623" watchObservedRunningTime="2026-10-29 22:22:57.310237648 +0000 UTC m=+439.791555142" Oct 29 22:22:57.333912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.333883 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" Oct 29 22:22:57.626556 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:57.626484 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js"] Oct 29 22:22:57.630337 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:22:57.630298 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f14e09a_672a_4d00_b240_c940f41939a6.slice/crio-e8c5cb43c1aedcbd8337554a9b8af8b7c75bf42f24ebc7daf84321e6401467c1 WatchSource:0}: Error finding container e8c5cb43c1aedcbd8337554a9b8af8b7c75bf42f24ebc7daf84321e6401467c1: Status 404 returned error can't find the container with id e8c5cb43c1aedcbd8337554a9b8af8b7c75bf42f24ebc7daf84321e6401467c1 Oct 29 22:22:58.244729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:58.244687 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" event={"ID":"1f14e09a-672a-4d00-b240-c940f41939a6","Type":"ContainerStarted","Data":"e8c5cb43c1aedcbd8337554a9b8af8b7c75bf42f24ebc7daf84321e6401467c1"} Oct 29 22:22:58.666286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:58.666193 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:58.818791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:58.818759 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:22:59.254026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:59.253934 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" event={"ID":"1f14e09a-672a-4d00-b240-c940f41939a6","Type":"ContainerStarted","Data":"74eeda73a386090864fea7e682f2e62beae4c84dc2d5d54e4a26a3e4cda0ad6b"} Oct 29 22:22:59.254026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:59.254024 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" Oct 29 22:22:59.289050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:59.289002 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" podStartSLOduration=1.966916414 podStartE2EDuration="3.288987267s" totalImagesPullingTime="1.322070853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:22:59.288569845 +0000 UTC m=+441.769887338" watchObservedRunningTime="2026-10-29 22:22:59.288987267 +0000 UTC m=+441.770304760" Oct 29 22:22:59.823528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:22:59.823498 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:00.050312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:00.050279 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:02.476305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:02.476272 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:02.619487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:02.619456 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:02.875574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:02.875497 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:03.012706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:03.012678 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:03.216188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:03.216163 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:03.282549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:03.282525 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:03.443000 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:03.442963 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:08.247363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:08.247331 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-9g5gh" Oct 29 22:23:08.247842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:08.247433 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:09.736588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:09.736553 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm"] Oct 29 22:23:09.740128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:09.740109 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:09.766993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:09.766951 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm"] Oct 29 22:23:09.885520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:09.885488 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43b51a0a-9fb9-4400-a434-74ce00751170-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-z7cwm\" (UID: \"43b51a0a-9fb9-4400-a434-74ce00751170\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:09.885670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:09.885531 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89b2r\" (UniqueName: \"kubernetes.io/projected/43b51a0a-9fb9-4400-a434-74ce00751170-kube-api-access-89b2r\") pod \"kuadrant-operator-controller-manager-6b76784d48-z7cwm\" (UID: \"43b51a0a-9fb9-4400-a434-74ce00751170\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:09.986479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:09.986445 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43b51a0a-9fb9-4400-a434-74ce00751170-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-z7cwm\" (UID: \"43b51a0a-9fb9-4400-a434-74ce00751170\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:09.986657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:09.986490 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89b2r\" (UniqueName: \"kubernetes.io/projected/43b51a0a-9fb9-4400-a434-74ce00751170-kube-api-access-89b2r\") pod \"kuadrant-operator-controller-manager-6b76784d48-z7cwm\" (UID: \"43b51a0a-9fb9-4400-a434-74ce00751170\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:09.986854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:09.986836 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43b51a0a-9fb9-4400-a434-74ce00751170-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-z7cwm\" (UID: \"43b51a0a-9fb9-4400-a434-74ce00751170\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:10.003927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:10.003905 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89b2r\" (UniqueName: \"kubernetes.io/projected/43b51a0a-9fb9-4400-a434-74ce00751170-kube-api-access-89b2r\") pod \"kuadrant-operator-controller-manager-6b76784d48-z7cwm\" (UID: \"43b51a0a-9fb9-4400-a434-74ce00751170\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:10.053010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:10.052982 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:10.260784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:10.260701 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-6p7js" Oct 29 22:23:10.357423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:10.357263 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm"] Oct 29 22:23:11.175857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.175824 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:11.306823 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.306794 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" event={"ID":"43b51a0a-9fb9-4400-a434-74ce00751170","Type":"ContainerStarted","Data":"b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925"} Oct 29 22:23:11.307010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.306866 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" event={"ID":"43b51a0a-9fb9-4400-a434-74ce00751170","Type":"ContainerStarted","Data":"f5724cc0a08bbe5b0cabdf21ea18733a72d37945445fb0397004b3316e3eb7cb"} Oct 29 22:23:11.307010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.306907 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:11.324809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.324785 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:11.368894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.368852 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" podStartSLOduration=2.368837149 podStartE2EDuration="2.368837149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:11.358622644 +0000 UTC m=+453.839940136" watchObservedRunningTime="2026-10-29 22:23:11.368837149 +0000 UTC m=+453.850154642" Oct 29 22:23:11.835562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.835531 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd"] Oct 29 22:23:11.835758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.835733 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" containerName="manager" containerID="cri-o://79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091" gracePeriod=2 Oct 29 22:23:11.845747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.845723 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd"] Oct 29 22:23:11.846978 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:11.846939 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:11.864239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.864213 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm"] Oct 29 22:23:11.868066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.868040 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs"] Oct 29 22:23:11.868647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.868624 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" containerName="manager" Oct 29 22:23:11.868647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.868646 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" containerName="manager" Oct 29 22:23:11.868799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.868727 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" containerName="manager" Oct 29 22:23:11.871953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.871934 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:11.877608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.877581 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm"] Oct 29 22:23:11.910975 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.910936 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4"] Oct 29 22:23:11.911449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.911427 5610 cpu_manager.go:414] "containerMap: removing container" podUID="43b51a0a-9fb9-4400-a434-74ce00751170" containerName="manager" Oct 29 22:23:11.911449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.911447 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b51a0a-9fb9-4400-a434-74ce00751170" containerName="manager" Oct 29 22:23:11.911624 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.911533 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b51a0a-9fb9-4400-a434-74ce00751170" containerName="manager" Oct 29 22:23:11.914697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.914679 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:11.919738 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.919710 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs"] Oct 29 22:23:11.926600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.926574 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw"] Oct 29 22:23:11.926864 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.926821 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" containerName="manager" containerID="cri-o://4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d" gracePeriod=2 Oct 29 22:23:11.951271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.951211 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw"] Oct 29 22:23:11.951972 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.951337 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4"] Oct 29 22:23:11.952178 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:11.952154 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:11.963632 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:11.963597 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:11.975816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.975785 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm"] Oct 29 22:23:11.976587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.976328 5610 cpu_manager.go:414] "containerMap: removing container" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" containerName="manager" Oct 29 22:23:11.976587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.976352 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" containerName="manager" Oct 29 22:23:11.976587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.976485 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" containerName="manager" Oct 29 22:23:11.979761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:11.979743 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" Oct 29 22:23:12.002983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.002941 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm"] Oct 29 22:23:12.002983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.002958 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4z2\" (UniqueName: \"kubernetes.io/projected/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-kube-api-access-hm4z2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lpdxs\" (UID: \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:12.003180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.003000 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lpdxs\" (UID: \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:12.003180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.003136 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h8rl\" (UniqueName: \"kubernetes.io/projected/f5b8074e-2730-4209-98c8-bd80202246d1-kube-api-access-5h8rl\") pod \"kuadrant-operator-controller-manager-6b76784d48-9bps4\" (UID: \"f5b8074e-2730-4209-98c8-bd80202246d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:12.003436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.003255 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b8074e-2730-4209-98c8-bd80202246d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-9bps4\" (UID: \"f5b8074e-2730-4209-98c8-bd80202246d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:12.017402 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.017357 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:12.060927 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.060855 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:12.071996 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.071958 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:12.103928 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.103895 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h8rl\" (UniqueName: \"kubernetes.io/projected/f5b8074e-2730-4209-98c8-bd80202246d1-kube-api-access-5h8rl\") pod \"kuadrant-operator-controller-manager-6b76784d48-9bps4\" (UID: \"f5b8074e-2730-4209-98c8-bd80202246d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:12.104072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.103936 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7cw\" (UniqueName: \"kubernetes.io/projected/eb2cab26-c089-4dfb-b58f-b778174e62a8-kube-api-access-gt7cw\") pod \"limitador-operator-controller-manager-856dd584c5-dtrnm\" (UID: \"eb2cab26-c089-4dfb-b58f-b778174e62a8\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" Oct 29 22:23:12.104072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.103971 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b8074e-2730-4209-98c8-bd80202246d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-9bps4\" (UID: \"f5b8074e-2730-4209-98c8-bd80202246d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:12.104072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.104024 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4z2\" (UniqueName: \"kubernetes.io/projected/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-kube-api-access-hm4z2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lpdxs\" (UID: \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:12.104072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.104055 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lpdxs\" (UID: \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:12.104716 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.104557 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lpdxs\" (UID: \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:12.105049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.105022 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b8074e-2730-4209-98c8-bd80202246d1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-9bps4\" (UID: \"f5b8074e-2730-4209-98c8-bd80202246d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:12.122786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.122759 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4z2\" (UniqueName: \"kubernetes.io/projected/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-kube-api-access-hm4z2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-lpdxs\" (UID: \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:12.123090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.123067 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h8rl\" (UniqueName: \"kubernetes.io/projected/f5b8074e-2730-4209-98c8-bd80202246d1-kube-api-access-5h8rl\") pod \"kuadrant-operator-controller-manager-6b76784d48-9bps4\" (UID: \"f5b8074e-2730-4209-98c8-bd80202246d1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:12.161712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.161693 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:12.168039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.168020 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:12.172755 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.172731 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:12.184184 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.184159 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:12.194960 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.194932 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:12.204818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.204788 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7cw\" (UniqueName: \"kubernetes.io/projected/eb2cab26-c089-4dfb-b58f-b778174e62a8-kube-api-access-gt7cw\") pod \"limitador-operator-controller-manager-856dd584c5-dtrnm\" (UID: \"eb2cab26-c089-4dfb-b58f-b778174e62a8\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" Oct 29 22:23:12.205788 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.205764 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:12.221969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.221947 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7cw\" (UniqueName: \"kubernetes.io/projected/eb2cab26-c089-4dfb-b58f-b778174e62a8-kube-api-access-gt7cw\") pod \"limitador-operator-controller-manager-856dd584c5-dtrnm\" (UID: \"eb2cab26-c089-4dfb-b58f-b778174e62a8\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" Oct 29 22:23:12.236280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.236258 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:12.242172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.242155 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:12.306020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.305162 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-kube-api-access-svqtl\" (UniqueName: \"kubernetes.io/projected/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-kube-api-access-svqtl\") pod \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\" (UID: \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\") " Oct 29 22:23:12.306020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.305267 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bc058d-2cfa-4033-ba92-45ddfe6999f3-kube-api-access-cxwxm\" (UniqueName: \"kubernetes.io/projected/21bc058d-2cfa-4033-ba92-45ddfe6999f3-kube-api-access-cxwxm\") pod \"21bc058d-2cfa-4033-ba92-45ddfe6999f3\" (UID: \"21bc058d-2cfa-4033-ba92-45ddfe6999f3\") " Oct 29 22:23:12.306020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.305329 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-extensions-socket-volume\") pod \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\" (UID: \"1b76c04f-c8b7-4917-8a0f-5dca81eabb31\") " Oct 29 22:23:12.306238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.306120 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-extensions-socket-volume" pod "1b76c04f-c8b7-4917-8a0f-5dca81eabb31" (UID: "1b76c04f-c8b7-4917-8a0f-5dca81eabb31"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:12.309411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.309178 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bc058d-2cfa-4033-ba92-45ddfe6999f3-kube-api-access-cxwxm" pod "21bc058d-2cfa-4033-ba92-45ddfe6999f3" (UID: "21bc058d-2cfa-4033-ba92-45ddfe6999f3"). InnerVolumeSpecName "kube-api-access-cxwxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:12.309411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.309243 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-kube-api-access-svqtl" pod "1b76c04f-c8b7-4917-8a0f-5dca81eabb31" (UID: "1b76c04f-c8b7-4917-8a0f-5dca81eabb31"). InnerVolumeSpecName "kube-api-access-svqtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:12.313467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.313427 5610 generic.go:356] "Generic (PLEG): container finished" podID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" containerID="4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d" exitCode=0 Oct 29 22:23:12.313467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.313604 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:12.314041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.314023 5610 scope.go:122] "RemoveContainer" containerID="4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d" Oct 29 22:23:12.314452 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.314434 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" Oct 29 22:23:12.317352 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.317217 5610 generic.go:356] "Generic (PLEG): container finished" podID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" containerID="79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091" exitCode=0 Oct 29 22:23:12.317352 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.317320 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:12.328833 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.327114 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:12.332251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.331362 5610 scope.go:122] "RemoveContainer" containerID="4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d" Oct 29 22:23:12.332251 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.331804 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d\": container with ID starting with 4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d not found: ID does not exist" containerID="4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d" Oct 29 22:23:12.332251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.331840 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d"} err="failed to get container status \"4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d\": rpc error: code = NotFound desc = could not find container \"4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d\": container with ID starting with 4743278aba20ad5e78717dbd284111e7039609d7b0942fb2017be8b213b70b0d not found: ID does not exist" Oct 29 22:23:12.332251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.331866 5610 scope.go:122] "RemoveContainer" containerID="79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091" Oct 29 22:23:12.338486 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.338456 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:12.342407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.342359 5610 scope.go:122] "RemoveContainer" containerID="79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091" Oct 29 22:23:12.342714 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.342664 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091\": container with ID starting with 79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091 not found: ID does not exist" containerID="79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091" Oct 29 22:23:12.342714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.342690 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091"} err="failed to get container status \"79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091\": rpc error: code = NotFound desc = could not find container \"79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091\": container with ID starting with 79ec511d9d04a7fe72939b36e1e872e0a49bd59581259de97c5a93d11c056091 not found: ID does not exist" Oct 29 22:23:12.350031 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.349869 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:12.362019 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:12.361985 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:12.406951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.406923 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svqtl\" (UniqueName: \"kubernetes.io/projected/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-kube-api-access-svqtl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:12.406951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.406952 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxwxm\" (UniqueName: \"kubernetes.io/projected/21bc058d-2cfa-4033-ba92-45ddfe6999f3-kube-api-access-cxwxm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:12.407117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.406962 5610 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b76c04f-c8b7-4917-8a0f-5dca81eabb31-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:12.564184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.564151 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4"] Oct 29 22:23:12.567437 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:23:12.567373 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b8074e_2730_4209_98c8_bd80202246d1.slice/crio-1594e33f80fc608c9c13115f4b4424d842407f5e4814281f6f79ee040bc21939 WatchSource:0}: Error finding container 1594e33f80fc608c9c13115f4b4424d842407f5e4814281f6f79ee040bc21939: Status 404 returned error can't find the container with id 1594e33f80fc608c9c13115f4b4424d842407f5e4814281f6f79ee040bc21939 Oct 29 22:23:12.567437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.567417 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs"] Oct 29 22:23:12.570333 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:23:12.570314 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615a8c22_82aa_45ae_b8ae_eee1e3f87d29.slice/crio-a32e3fa5d9eb36aba05c16614542314275bab707727bbf52f204027c7e9f5383 WatchSource:0}: Error finding container a32e3fa5d9eb36aba05c16614542314275bab707727bbf52f204027c7e9f5383: Status 404 returned error can't find the container with id a32e3fa5d9eb36aba05c16614542314275bab707727bbf52f204027c7e9f5383 Oct 29 22:23:12.626256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:12.624159 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm"] Oct 29 22:23:12.630688 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:23:12.630658 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2cab26_c089_4dfb_b58f_b778174e62a8.slice/crio-d4230a4e3c2be3ebf1d4d91c3e3a50d857dd4d612f6a5ffe3080ceb8a1671dd3 WatchSource:0}: Error finding container d4230a4e3c2be3ebf1d4d91c3e3a50d857dd4d612f6a5ffe3080ceb8a1671dd3: Status 404 returned error can't find the container with id d4230a4e3c2be3ebf1d4d91c3e3a50d857dd4d612f6a5ffe3080ceb8a1671dd3 Oct 29 22:23:13.328537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.328502 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" event={"ID":"615a8c22-82aa-45ae-b8ae-eee1e3f87d29","Type":"ContainerStarted","Data":"ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46"} Oct 29 22:23:13.329010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.328581 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" event={"ID":"615a8c22-82aa-45ae-b8ae-eee1e3f87d29","Type":"ContainerStarted","Data":"a32e3fa5d9eb36aba05c16614542314275bab707727bbf52f204027c7e9f5383"} Oct 29 22:23:13.329010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.328619 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:13.330341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.330322 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" event={"ID":"f5b8074e-2730-4209-98c8-bd80202246d1","Type":"ContainerStarted","Data":"da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927"} Oct 29 22:23:13.330464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.330359 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:13.330464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.330374 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" event={"ID":"f5b8074e-2730-4209-98c8-bd80202246d1","Type":"ContainerStarted","Data":"1594e33f80fc608c9c13115f4b4424d842407f5e4814281f6f79ee040bc21939"} Oct 29 22:23:13.333541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.333512 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" event={"ID":"eb2cab26-c089-4dfb-b58f-b778174e62a8","Type":"ContainerStarted","Data":"a2a6ccf24d4c73123d2ad7a062d9c10c638f9a42a61b91c9d272eb365b8cc666"} Oct 29 22:23:13.333613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.333557 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" event={"ID":"eb2cab26-c089-4dfb-b58f-b778174e62a8","Type":"ContainerStarted","Data":"d4230a4e3c2be3ebf1d4d91c3e3a50d857dd4d612f6a5ffe3080ceb8a1671dd3"} Oct 29 22:23:13.333665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.333624 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" Oct 29 22:23:13.333768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.333744 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" podUID="43b51a0a-9fb9-4400-a434-74ce00751170" containerName="manager" containerID="cri-o://b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925" gracePeriod=2 Oct 29 22:23:13.340323 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.340294 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:13.388130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.388082 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" podStartSLOduration=2.388066667 podStartE2EDuration="2.388066667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:13.387595898 +0000 UTC m=+455.868913391" watchObservedRunningTime="2026-10-29 22:23:13.388066667 +0000 UTC m=+455.869384161" Oct 29 22:23:13.398660 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.398621 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:13.434825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.434776 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" podStartSLOduration=2.434762818 podStartE2EDuration="2.434762818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:13.434277604 +0000 UTC m=+455.915595094" watchObservedRunningTime="2026-10-29 22:23:13.434762818 +0000 UTC m=+455.916080310" Oct 29 22:23:13.445502 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.445469 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-z7cwm\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="43b51a0a-9fb9-4400-a434-74ce00751170" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:13.456578 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.456545 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:13.467683 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.467643 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:13.508388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.508338 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" podStartSLOduration=2.508324274 podStartE2EDuration="2.508324274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:13.507906713 +0000 UTC m=+455.989224208" watchObservedRunningTime="2026-10-29 22:23:13.508324274 +0000 UTC m=+455.989641767" Oct 29 22:23:13.585869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.585804 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:13.597155 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.597125 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rt2bw\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rt2bw" Oct 29 22:23:13.608098 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.608070 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-z7cwm\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="43b51a0a-9fb9-4400-a434-74ce00751170" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:13.618917 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.618895 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-2dgcd\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-2dgcd" Oct 29 22:23:13.624239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.624206 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:13.633373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.633349 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b76c04f-c8b7-4917-8a0f-5dca81eabb31" path="/var/lib/kubelet/pods/1b76c04f-c8b7-4917-8a0f-5dca81eabb31/volumes" Oct 29 22:23:13.633731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.633717 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21bc058d-2cfa-4033-ba92-45ddfe6999f3" path="/var/lib/kubelet/pods/21bc058d-2cfa-4033-ba92-45ddfe6999f3/volumes" Oct 29 22:23:13.708129 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:13.708090 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:23:13.721359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.721335 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b51a0a-9fb9-4400-a434-74ce00751170-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43b51a0a-9fb9-4400-a434-74ce00751170-extensions-socket-volume\") pod \"43b51a0a-9fb9-4400-a434-74ce00751170\" (UID: \"43b51a0a-9fb9-4400-a434-74ce00751170\") " Oct 29 22:23:13.721526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.721365 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b51a0a-9fb9-4400-a434-74ce00751170-kube-api-access-89b2r\" (UniqueName: \"kubernetes.io/projected/43b51a0a-9fb9-4400-a434-74ce00751170-kube-api-access-89b2r\") pod \"43b51a0a-9fb9-4400-a434-74ce00751170\" (UID: \"43b51a0a-9fb9-4400-a434-74ce00751170\") " Oct 29 22:23:13.721776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.721754 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b51a0a-9fb9-4400-a434-74ce00751170-extensions-socket-volume" pod "43b51a0a-9fb9-4400-a434-74ce00751170" (UID: "43b51a0a-9fb9-4400-a434-74ce00751170"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:13.723307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.723287 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b51a0a-9fb9-4400-a434-74ce00751170-kube-api-access-89b2r" pod "43b51a0a-9fb9-4400-a434-74ce00751170" (UID: "43b51a0a-9fb9-4400-a434-74ce00751170"). InnerVolumeSpecName "kube-api-access-89b2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:13.761061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.761025 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:13.822590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.822558 5610 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43b51a0a-9fb9-4400-a434-74ce00751170-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:13.822590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:13.822585 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89b2r\" (UniqueName: \"kubernetes.io/projected/43b51a0a-9fb9-4400-a434-74ce00751170-kube-api-access-89b2r\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:14.338540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:14.338509 5610 generic.go:356] "Generic (PLEG): container finished" podID="43b51a0a-9fb9-4400-a434-74ce00751170" containerID="b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925" exitCode=0 Oct 29 22:23:14.338945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:14.338550 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-z7cwm" Oct 29 22:23:14.338945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:14.338599 5610 scope.go:122] "RemoveContainer" containerID="b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925" Oct 29 22:23:14.347187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:14.347161 5610 scope.go:122] "RemoveContainer" containerID="b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925" Oct 29 22:23:14.347476 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:14.347459 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925\": container with ID starting with b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925 not found: ID does not exist" containerID="b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925" Oct 29 22:23:14.347517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:14.347485 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925"} err="failed to get container status \"b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925\": rpc error: code = NotFound desc = could not find container \"b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925\": container with ID starting with b18201663839b0aec6b6d2e5c928d79e23d5579e5b9db27e742dd5f556de3925 not found: ID does not exist" Oct 29 22:23:15.633884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:15.633850 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b51a0a-9fb9-4400-a434-74ce00751170" path="/var/lib/kubelet/pods/43b51a0a-9fb9-4400-a434-74ce00751170/volumes" Oct 29 22:23:17.948531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:17.948473 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:18.085165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:18.085134 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:21.777659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:21.777583 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:21.917485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:21.917446 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:22.059046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:22.058953 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:22.951062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:22.951032 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:23.091318 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:23.091275 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:24.341196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.341167 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:24.341626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.341242 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dtrnm" Oct 29 22:23:24.341626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.341275 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:24.428919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.428886 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs"] Oct 29 22:23:24.429144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.429094 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" podUID="615a8c22-82aa-45ae-b8ae-eee1e3f87d29" containerName="manager" containerID="cri-o://ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46" gracePeriod=10 Oct 29 22:23:24.714626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.714594 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw"] Oct 29 22:23:24.732558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.732522 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:24.736281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.736259 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:24.745224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.745200 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw"] Oct 29 22:23:24.816694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.816662 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-kube-api-access-hm4z2\" (UniqueName: \"kubernetes.io/projected/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-kube-api-access-hm4z2\") pod \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\" (UID: \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\") " Oct 29 22:23:24.816862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.816752 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-extensions-socket-volume\") pod \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\" (UID: \"615a8c22-82aa-45ae-b8ae-eee1e3f87d29\") " Oct 29 22:23:24.816918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.816892 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5xj\" (UniqueName: \"kubernetes.io/projected/70fdbd02-4506-4534-a9b0-95ebee3b33b5-kube-api-access-bv5xj\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5shkw\" (UID: \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:24.817068 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.817032 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70fdbd02-4506-4534-a9b0-95ebee3b33b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5shkw\" (UID: \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:24.817198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.817129 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-extensions-socket-volume" pod "615a8c22-82aa-45ae-b8ae-eee1e3f87d29" (UID: "615a8c22-82aa-45ae-b8ae-eee1e3f87d29"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:24.820011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.819983 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-kube-api-access-hm4z2" pod "615a8c22-82aa-45ae-b8ae-eee1e3f87d29" (UID: "615a8c22-82aa-45ae-b8ae-eee1e3f87d29"). InnerVolumeSpecName "kube-api-access-hm4z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:24.917576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.917487 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70fdbd02-4506-4534-a9b0-95ebee3b33b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5shkw\" (UID: \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:24.917576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.917544 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5xj\" (UniqueName: \"kubernetes.io/projected/70fdbd02-4506-4534-a9b0-95ebee3b33b5-kube-api-access-bv5xj\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5shkw\" (UID: \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:24.917576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.917579 5610 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:24.917877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.917590 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4z2\" (UniqueName: \"kubernetes.io/projected/615a8c22-82aa-45ae-b8ae-eee1e3f87d29-kube-api-access-hm4z2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:24.917969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.917938 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70fdbd02-4506-4534-a9b0-95ebee3b33b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5shkw\" (UID: \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:24.935847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:24.935822 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5xj\" (UniqueName: \"kubernetes.io/projected/70fdbd02-4506-4534-a9b0-95ebee3b33b5-kube-api-access-bv5xj\") pod \"kuadrant-operator-controller-manager-57bb9bfff-5shkw\" (UID: \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:25.073076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.073045 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:25.388759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.388717 5610 generic.go:356] "Generic (PLEG): container finished" podID="615a8c22-82aa-45ae-b8ae-eee1e3f87d29" containerID="ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46" exitCode=0 Oct 29 22:23:25.389181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.388781 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" event={"ID":"615a8c22-82aa-45ae-b8ae-eee1e3f87d29","Type":"ContainerDied","Data":"ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46"} Oct 29 22:23:25.389181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.388783 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" Oct 29 22:23:25.389181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.388870 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs" event={"ID":"615a8c22-82aa-45ae-b8ae-eee1e3f87d29","Type":"ContainerDied","Data":"a32e3fa5d9eb36aba05c16614542314275bab707727bbf52f204027c7e9f5383"} Oct 29 22:23:25.389181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.388901 5610 scope.go:122] "RemoveContainer" containerID="ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46" Oct 29 22:23:25.397860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.397842 5610 scope.go:122] "RemoveContainer" containerID="ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46" Oct 29 22:23:25.398140 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:25.398119 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46\": container with ID starting with ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46 not found: ID does not exist" containerID="ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46" Oct 29 22:23:25.398230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.398146 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46"} err="failed to get container status \"ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46\": rpc error: code = NotFound desc = could not find container \"ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46\": container with ID starting with ff8b59fb7fa05db8de860bf086516375875e9f46da20d78b721644dd2c37fa46 not found: ID does not exist" Oct 29 22:23:25.409927 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:23:25.409908 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fdbd02_4506_4534_a9b0_95ebee3b33b5.slice/crio-5ad7dfa7130e68e7c51ef50fa65c72b1b96de9471179283100c221651250968e WatchSource:0}: Error finding container 5ad7dfa7130e68e7c51ef50fa65c72b1b96de9471179283100c221651250968e: Status 404 returned error can't find the container with id 5ad7dfa7130e68e7c51ef50fa65c72b1b96de9471179283100c221651250968e Oct 29 22:23:25.416211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.416188 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw"] Oct 29 22:23:25.459146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.459125 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs"] Oct 29 22:23:25.468902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.468882 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-lpdxs"] Oct 29 22:23:25.633651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:25.633620 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615a8c22-82aa-45ae-b8ae-eee1e3f87d29" path="/var/lib/kubelet/pods/615a8c22-82aa-45ae-b8ae-eee1e3f87d29/volumes" Oct 29 22:23:26.395949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:26.395916 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" event={"ID":"70fdbd02-4506-4534-a9b0-95ebee3b33b5","Type":"ContainerStarted","Data":"e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e"} Oct 29 22:23:26.396350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:26.395965 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" event={"ID":"70fdbd02-4506-4534-a9b0-95ebee3b33b5","Type":"ContainerStarted","Data":"5ad7dfa7130e68e7c51ef50fa65c72b1b96de9471179283100c221651250968e"} Oct 29 22:23:26.396350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:26.395997 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:26.453264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:26.453213 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" podStartSLOduration=2.453199281 podStartE2EDuration="2.453199281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:23:26.442679535 +0000 UTC m=+468.923997055" watchObservedRunningTime="2026-10-29 22:23:26.453199281 +0000 UTC m=+468.934516768" Oct 29 22:23:32.153605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:32.153568 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:32.306876 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:32.306846 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:34.597916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:34.597880 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:34.730221 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:34.730186 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:35.274133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:35.274106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:37.402696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.402659 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 22:23:37.481483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.481450 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4"] Oct 29 22:23:37.481757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.481727 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" podUID="f5b8074e-2730-4209-98c8-bd80202246d1" containerName="manager" containerID="cri-o://da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927" gracePeriod=10 Oct 29 22:23:37.725176 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.725152 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:37.822818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.822783 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b8074e-2730-4209-98c8-bd80202246d1-kube-api-access-5h8rl\" (UniqueName: \"kubernetes.io/projected/f5b8074e-2730-4209-98c8-bd80202246d1-kube-api-access-5h8rl\") pod \"f5b8074e-2730-4209-98c8-bd80202246d1\" (UID: \"f5b8074e-2730-4209-98c8-bd80202246d1\") " Oct 29 22:23:37.822818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.822819 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b8074e-2730-4209-98c8-bd80202246d1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b8074e-2730-4209-98c8-bd80202246d1-extensions-socket-volume\") pod \"f5b8074e-2730-4209-98c8-bd80202246d1\" (UID: \"f5b8074e-2730-4209-98c8-bd80202246d1\") " Oct 29 22:23:37.823231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.823211 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b8074e-2730-4209-98c8-bd80202246d1-extensions-socket-volume" pod "f5b8074e-2730-4209-98c8-bd80202246d1" (UID: "f5b8074e-2730-4209-98c8-bd80202246d1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:23:37.824842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.824819 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b8074e-2730-4209-98c8-bd80202246d1-kube-api-access-5h8rl" pod "f5b8074e-2730-4209-98c8-bd80202246d1" (UID: "f5b8074e-2730-4209-98c8-bd80202246d1"). InnerVolumeSpecName "kube-api-access-5h8rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:23:37.923845 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.923772 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h8rl\" (UniqueName: \"kubernetes.io/projected/f5b8074e-2730-4209-98c8-bd80202246d1-kube-api-access-5h8rl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:37.923845 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:37.923797 5610 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5b8074e-2730-4209-98c8-bd80202246d1-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:23:38.451899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.451865 5610 generic.go:356] "Generic (PLEG): container finished" podID="f5b8074e-2730-4209-98c8-bd80202246d1" containerID="da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927" exitCode=0 Oct 29 22:23:38.452310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.451911 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" Oct 29 22:23:38.452310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.451930 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" event={"ID":"f5b8074e-2730-4209-98c8-bd80202246d1","Type":"ContainerDied","Data":"da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927"} Oct 29 22:23:38.452310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.451977 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4" event={"ID":"f5b8074e-2730-4209-98c8-bd80202246d1","Type":"ContainerDied","Data":"1594e33f80fc608c9c13115f4b4424d842407f5e4814281f6f79ee040bc21939"} Oct 29 22:23:38.452310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.452002 5610 scope.go:122] "RemoveContainer" containerID="da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927" Oct 29 22:23:38.463417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.463221 5610 scope.go:122] "RemoveContainer" containerID="da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927" Oct 29 22:23:38.463641 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:38.463614 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927\": container with ID starting with da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927 not found: ID does not exist" containerID="da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927" Oct 29 22:23:38.463752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.463650 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927"} err="failed to get container status \"da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927\": rpc error: code = NotFound desc = could not find container \"da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927\": container with ID starting with da8a5a30abdf5c9c5fc0e0f7111873df44dcf3e07674b155c654190aca402927 not found: ID does not exist" Oct 29 22:23:38.497993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.497965 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4"] Oct 29 22:23:38.501952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:38.501932 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-9bps4"] Oct 29 22:23:39.633479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:39.633444 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b8074e-2730-4209-98c8-bd80202246d1" path="/var/lib/kubelet/pods/f5b8074e-2730-4209-98c8-bd80202246d1/volumes" Oct 29 22:23:41.714406 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:41.714357 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:41.935251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:41.935216 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:42.535627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:42.535600 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:43.708101 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:23:43.708066 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:23:57.704308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:57.704276 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:57.840229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:57.840167 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:59.051306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.051271 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7r22r"] Oct 29 22:23:59.051722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.051681 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f5b8074e-2730-4209-98c8-bd80202246d1" containerName="manager" Oct 29 22:23:59.051722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.051695 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b8074e-2730-4209-98c8-bd80202246d1" containerName="manager" Oct 29 22:23:59.051722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.051721 5610 cpu_manager.go:414] "containerMap: removing container" podUID="615a8c22-82aa-45ae-b8ae-eee1e3f87d29" containerName="manager" Oct 29 22:23:59.051836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.051725 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615a8c22-82aa-45ae-b8ae-eee1e3f87d29" containerName="manager" Oct 29 22:23:59.051836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.051774 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b8074e-2730-4209-98c8-bd80202246d1" containerName="manager" Oct 29 22:23:59.051836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.051785 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="615a8c22-82aa-45ae-b8ae-eee1e3f87d29" containerName="manager" Oct 29 22:23:59.054675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.054655 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:23:59.066788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.066768 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 29 22:23:59.076777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.076756 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kzfft\"" Oct 29 22:23:59.078064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.078044 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7r22r"] Oct 29 22:23:59.094197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.094173 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:59.180989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.180955 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7r22r"] Oct 29 22:23:59.203759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.203735 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b31c7e9-5cfd-43aa-a80d-c930022437d7-config-file\") pod \"limitador-limitador-74dfcfb894-7r22r\" (UID: \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:23:59.203927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.203892 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4wh\" (UniqueName: \"kubernetes.io/projected/3b31c7e9-5cfd-43aa-a80d-c930022437d7-kube-api-access-5w4wh\") pod \"limitador-limitador-74dfcfb894-7r22r\" (UID: \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:23:59.222172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.222145 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:59.310774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.310696 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4wh\" (UniqueName: \"kubernetes.io/projected/3b31c7e9-5cfd-43aa-a80d-c930022437d7-kube-api-access-5w4wh\") pod \"limitador-limitador-74dfcfb894-7r22r\" (UID: \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:23:59.310774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.310753 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b31c7e9-5cfd-43aa-a80d-c930022437d7-config-file\") pod \"limitador-limitador-74dfcfb894-7r22r\" (UID: \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:23:59.311458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.311438 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b31c7e9-5cfd-43aa-a80d-c930022437d7-config-file\") pod \"limitador-limitador-74dfcfb894-7r22r\" (UID: \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:23:59.327552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.327524 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4wh\" (UniqueName: \"kubernetes.io/projected/3b31c7e9-5cfd-43aa-a80d-c930022437d7-kube-api-access-5w4wh\") pod \"limitador-limitador-74dfcfb894-7r22r\" (UID: \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:23:59.350386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.350358 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:23:59.366423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.366384 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:23:59.653806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:23:59.653736 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7r22r"] Oct 29 22:23:59.657012 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:23:59.656984 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b31c7e9_5cfd_43aa_a80d_c930022437d7.slice/crio-895d6f6c752be6e7a2b397b037b9d8541679125e94d2d6f7c53001ca56682763 WatchSource:0}: Error finding container 895d6f6c752be6e7a2b397b037b9d8541679125e94d2d6f7c53001ca56682763: Status 404 returned error can't find the container with id 895d6f6c752be6e7a2b397b037b9d8541679125e94d2d6f7c53001ca56682763 Oct 29 22:24:00.011981 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:00.011949 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:00.141289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:00.141258 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:00.552841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:00.552804 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" event={"ID":"3b31c7e9-5cfd-43aa-a80d-c930022437d7","Type":"ContainerStarted","Data":"895d6f6c752be6e7a2b397b037b9d8541679125e94d2d6f7c53001ca56682763"} Oct 29 22:24:01.558749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:01.558713 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" event={"ID":"3b31c7e9-5cfd-43aa-a80d-c930022437d7","Type":"ContainerStarted","Data":"d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2"} Oct 29 22:24:01.559150 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:01.558802 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:24:01.592677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:01.592629 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" podStartSLOduration=1.451052962 podStartE2EDuration="2.592616036s" totalImagesPullingTime="1.141563074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:01.591345428 +0000 UTC m=+504.072662936" watchObservedRunningTime="2026-10-29 22:24:01.592616036 +0000 UTC m=+504.073933529" Oct 29 22:24:12.564565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:12.564537 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:24:13.707657 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:24:13.707616 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:24:14.583453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:14.583423 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7r22r"] Oct 29 22:24:14.583746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:14.583714 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" podUID="3b31c7e9-5cfd-43aa-a80d-c930022437d7" containerName="limitador" containerID="cri-o://d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2" gracePeriod=30 Oct 29 22:24:15.125557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.125534 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:24:15.146155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.146101 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b31c7e9-5cfd-43aa-a80d-c930022437d7-config-file\" (UniqueName: \"kubernetes.io/configmap/3b31c7e9-5cfd-43aa-a80d-c930022437d7-config-file\") pod \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\" (UID: \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\") " Oct 29 22:24:15.146155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.146139 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b31c7e9-5cfd-43aa-a80d-c930022437d7-kube-api-access-5w4wh\" (UniqueName: \"kubernetes.io/projected/3b31c7e9-5cfd-43aa-a80d-c930022437d7-kube-api-access-5w4wh\") pod \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\" (UID: \"3b31c7e9-5cfd-43aa-a80d-c930022437d7\") " Oct 29 22:24:15.146427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.146386 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b31c7e9-5cfd-43aa-a80d-c930022437d7-config-file" pod "3b31c7e9-5cfd-43aa-a80d-c930022437d7" (UID: "3b31c7e9-5cfd-43aa-a80d-c930022437d7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:24:15.148118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.148092 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b31c7e9-5cfd-43aa-a80d-c930022437d7-kube-api-access-5w4wh" pod "3b31c7e9-5cfd-43aa-a80d-c930022437d7" (UID: "3b31c7e9-5cfd-43aa-a80d-c930022437d7"). InnerVolumeSpecName "kube-api-access-5w4wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:15.246942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.246909 5610 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b31c7e9-5cfd-43aa-a80d-c930022437d7-config-file\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:24:15.246942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.246936 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4wh\" (UniqueName: \"kubernetes.io/projected/3b31c7e9-5cfd-43aa-a80d-c930022437d7-kube-api-access-5w4wh\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:24:15.618193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.618156 5610 generic.go:356] "Generic (PLEG): container finished" podID="3b31c7e9-5cfd-43aa-a80d-c930022437d7" containerID="d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2" exitCode=0 Oct 29 22:24:15.618358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.618205 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" Oct 29 22:24:15.618358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.618229 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" event={"ID":"3b31c7e9-5cfd-43aa-a80d-c930022437d7","Type":"ContainerDied","Data":"d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2"} Oct 29 22:24:15.618358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.618286 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7r22r" event={"ID":"3b31c7e9-5cfd-43aa-a80d-c930022437d7","Type":"ContainerDied","Data":"895d6f6c752be6e7a2b397b037b9d8541679125e94d2d6f7c53001ca56682763"} Oct 29 22:24:15.618358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.618308 5610 scope.go:122] "RemoveContainer" containerID="d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2" Oct 29 22:24:15.627210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.627195 5610 scope.go:122] "RemoveContainer" containerID="d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2" Oct 29 22:24:15.627490 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:24:15.627472 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2\": container with ID starting with d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2 not found: ID does not exist" containerID="d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2" Oct 29 22:24:15.627562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.627497 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2"} err="failed to get container status \"d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2\": rpc error: code = NotFound desc = could not find container \"d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2\": container with ID starting with d97ddd404ab0ca51100d4ae6bd921c791652f3dda66da486db14f432359d6fc2 not found: ID does not exist" Oct 29 22:24:15.663420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.663383 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7r22r"] Oct 29 22:24:15.672739 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:15.672716 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7r22r"] Oct 29 22:24:17.632549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:17.632517 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b31c7e9-5cfd-43aa-a80d-c930022437d7" path="/var/lib/kubelet/pods/3b31c7e9-5cfd-43aa-a80d-c930022437d7/volumes" Oct 29 22:24:21.118231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.118190 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:21.737312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.737273 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r"] Oct 29 22:24:21.737702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.737685 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3b31c7e9-5cfd-43aa-a80d-c930022437d7" containerName="limitador" Oct 29 22:24:21.737759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.737704 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b31c7e9-5cfd-43aa-a80d-c930022437d7" containerName="limitador" Oct 29 22:24:21.737800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.737782 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b31c7e9-5cfd-43aa-a80d-c930022437d7" containerName="limitador" Oct 29 22:24:21.742183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.742164 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:21.754595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.754571 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 29 22:24:21.754730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.754607 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 29 22:24:21.764474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.764453 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 29 22:24:21.764687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.764661 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rd7ph\"" Oct 29 22:24:21.765571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.765550 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r"] Oct 29 22:24:21.797255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.797228 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e63c0524-70e2-4969-a8bb-2ae37ceaf9b6-data\") pod \"postgres-7cdbf6dc44-n6z8r\" (UID: \"e63c0524-70e2-4969-a8bb-2ae37ceaf9b6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:21.797425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.797282 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgr5r\" (UniqueName: \"kubernetes.io/projected/e63c0524-70e2-4969-a8bb-2ae37ceaf9b6-kube-api-access-tgr5r\") pod \"postgres-7cdbf6dc44-n6z8r\" (UID: \"e63c0524-70e2-4969-a8bb-2ae37ceaf9b6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:21.898315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.898286 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e63c0524-70e2-4969-a8bb-2ae37ceaf9b6-data\") pod \"postgres-7cdbf6dc44-n6z8r\" (UID: \"e63c0524-70e2-4969-a8bb-2ae37ceaf9b6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:21.898487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.898335 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgr5r\" (UniqueName: \"kubernetes.io/projected/e63c0524-70e2-4969-a8bb-2ae37ceaf9b6-kube-api-access-tgr5r\") pod \"postgres-7cdbf6dc44-n6z8r\" (UID: \"e63c0524-70e2-4969-a8bb-2ae37ceaf9b6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:21.898712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.898695 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e63c0524-70e2-4969-a8bb-2ae37ceaf9b6-data\") pod \"postgres-7cdbf6dc44-n6z8r\" (UID: \"e63c0524-70e2-4969-a8bb-2ae37ceaf9b6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:21.915223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:21.915200 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgr5r\" (UniqueName: \"kubernetes.io/projected/e63c0524-70e2-4969-a8bb-2ae37ceaf9b6-kube-api-access-tgr5r\") pod \"postgres-7cdbf6dc44-n6z8r\" (UID: \"e63c0524-70e2-4969-a8bb-2ae37ceaf9b6\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:22.054632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:22.054556 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:22.344230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:22.344147 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r"] Oct 29 22:24:22.347861 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:24:22.347833 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63c0524_70e2_4969_a8bb_2ae37ceaf9b6.slice/crio-e0bbf8e6f0b74c538d17ef87475ef299ce1d5aa661dbf06c2b4b8617d9361594 WatchSource:0}: Error finding container e0bbf8e6f0b74c538d17ef87475ef299ce1d5aa661dbf06c2b4b8617d9361594: Status 404 returned error can't find the container with id e0bbf8e6f0b74c538d17ef87475ef299ce1d5aa661dbf06c2b4b8617d9361594 Oct 29 22:24:22.462174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:22.462142 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:22.648815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:22.648732 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" event={"ID":"e63c0524-70e2-4969-a8bb-2ae37ceaf9b6","Type":"ContainerStarted","Data":"e0bbf8e6f0b74c538d17ef87475ef299ce1d5aa661dbf06c2b4b8617d9361594"} Oct 29 22:24:27.672894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:27.672846 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" event={"ID":"e63c0524-70e2-4969-a8bb-2ae37ceaf9b6","Type":"ContainerStarted","Data":"50c8fb79f56a5968e16c173e02874488409441270dd7b0907f001baf74fe44a4"} Oct 29 22:24:27.673286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:27.672948 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:27.705624 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:27.705563 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" podStartSLOduration=2.167710779 podStartE2EDuration="6.705548699s" totalImagesPullingTime="4.53783792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:27.705140092 +0000 UTC m=+530.186457584" watchObservedRunningTime="2026-10-29 22:24:27.705548699 +0000 UTC m=+530.186866192" Oct 29 22:24:33.706101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:33.706071 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-n6z8r" Oct 29 22:24:34.196530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:34.196483 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:34.421658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:34.421620 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:35.395380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:35.395346 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:43.707612 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:24:43.707569 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:24:43.878476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.878438 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:43.889562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.889529 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-brs7d"] Oct 29 22:24:43.910257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.910232 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:43.913510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.913480 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-brs7d"] Oct 29 22:24:43.946464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.946431 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 29 22:24:43.959865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.959807 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 29 22:24:43.959974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.959953 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 29 22:24:43.959974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.959962 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bfzvw\"" Oct 29 22:24:43.985151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.985122 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:43.985265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.985171 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-webhook-cert\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:43.985308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:43.985291 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8gf\" (UniqueName: \"kubernetes.io/projected/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-kube-api-access-zq8gf\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.010138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.010113 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:44.086627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.086591 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8gf\" (UniqueName: \"kubernetes.io/projected/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-kube-api-access-zq8gf\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.086776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.086662 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.086776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.086704 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-webhook-cert\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.086894 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:24:44.086815 5610 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Oct 29 22:24:44.086894 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:24:44.086887 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs podName:99169dd1-cd7a-48bc-8b6b-4040ebecd14c nodeName:}" failed. No retries permitted until 2026-10-29 22:24:44.586870383 +0000 UTC m=+547.068187868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs") pod "maas-controller-577f596567-brs7d" (UID: "99169dd1-cd7a-48bc-8b6b-4040ebecd14c") : secret "maas-controller-metrics-tls" not found Oct 29 22:24:44.089191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.089166 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-webhook-cert\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.103432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.103411 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8gf\" (UniqueName: \"kubernetes.io/projected/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-kube-api-access-zq8gf\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.150643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.150615 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:44.276273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.276215 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86789cb564-phf64"] Oct 29 22:24:44.279830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.279810 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.289780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.289755 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:44.299480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.299460 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86789cb564-phf64"] Oct 29 22:24:44.388941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.388916 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-metrics-certs\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.389078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.388950 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-webhook-cert\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.389127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.389084 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bt8g\" (UniqueName: \"kubernetes.io/projected/f7f71636-1271-4e26-b3a4-cad6448fb72a-kube-api-access-5bt8g\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.489674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.489641 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bt8g\" (UniqueName: \"kubernetes.io/projected/f7f71636-1271-4e26-b3a4-cad6448fb72a-kube-api-access-5bt8g\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.489832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.489697 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-metrics-certs\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.489832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.489715 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-webhook-cert\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.492029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.492006 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-webhook-cert\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.492164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.492129 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-metrics-certs\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.505573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.505546 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bt8g\" (UniqueName: \"kubernetes.io/projected/f7f71636-1271-4e26-b3a4-cad6448fb72a-kube-api-access-5bt8g\") pod \"maas-controller-86789cb564-phf64\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.590151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.590094 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.592108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.592083 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs\") pod \"maas-controller-577f596567-brs7d\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.592209 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.592196 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:44.824899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.824869 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:44.882984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:44.882959 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86789cb564-phf64"] Oct 29 22:24:44.889282 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:24:44.889250 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f71636_1271_4e26_b3a4_cad6448fb72a.slice/crio-456c1eb390e8aada7bcb978c4892bbe9c90513c2f1b9c182ba00165414179c28 WatchSource:0}: Error finding container 456c1eb390e8aada7bcb978c4892bbe9c90513c2f1b9c182ba00165414179c28: Status 404 returned error can't find the container with id 456c1eb390e8aada7bcb978c4892bbe9c90513c2f1b9c182ba00165414179c28 Oct 29 22:24:45.114798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:45.114770 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-brs7d"] Oct 29 22:24:45.117802 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:24:45.117770 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99169dd1_cd7a_48bc_8b6b_4040ebecd14c.slice/crio-cd3673bbebc0d6badbfc0105caf7bae1ebb9771f81ac16cdd5c3fc34f7ec3709 WatchSource:0}: Error finding container cd3673bbebc0d6badbfc0105caf7bae1ebb9771f81ac16cdd5c3fc34f7ec3709: Status 404 returned error can't find the container with id cd3673bbebc0d6badbfc0105caf7bae1ebb9771f81ac16cdd5c3fc34f7ec3709 Oct 29 22:24:45.780133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:45.780100 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-brs7d" event={"ID":"99169dd1-cd7a-48bc-8b6b-4040ebecd14c","Type":"ContainerStarted","Data":"cd3673bbebc0d6badbfc0105caf7bae1ebb9771f81ac16cdd5c3fc34f7ec3709"} Oct 29 22:24:45.782259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:45.782221 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-phf64" event={"ID":"f7f71636-1271-4e26-b3a4-cad6448fb72a","Type":"ContainerStarted","Data":"456c1eb390e8aada7bcb978c4892bbe9c90513c2f1b9c182ba00165414179c28"} Oct 29 22:24:47.794415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:47.793917 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-phf64" event={"ID":"f7f71636-1271-4e26-b3a4-cad6448fb72a","Type":"ContainerStarted","Data":"0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140"} Oct 29 22:24:47.794415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:47.794000 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:47.830040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:47.829987 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-86789cb564-phf64" podStartSLOduration=1.031034216 podStartE2EDuration="3.829973105s" totalImagesPullingTime="2.798938889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:47.828944599 +0000 UTC m=+550.310262093" watchObservedRunningTime="2026-10-29 22:24:47.829973105 +0000 UTC m=+550.311290597" Oct 29 22:24:47.964727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:47.964672 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:48.109554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:48.109522 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:48.799150 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:48.799109 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-brs7d" event={"ID":"99169dd1-cd7a-48bc-8b6b-4040ebecd14c","Type":"ContainerStarted","Data":"8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86"} Oct 29 22:24:48.835312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:48.835267 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-brs7d" podStartSLOduration=2.976092592 podStartE2EDuration="5.835253915s" totalImagesPullingTime="2.859161323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:24:48.83344685 +0000 UTC m=+551.314764343" watchObservedRunningTime="2026-10-29 22:24:48.835253915 +0000 UTC m=+551.316571408" Oct 29 22:24:49.803033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:49.802964 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:50.731266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:50.731229 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:51.221505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.221479 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5"] Oct 29 22:24:51.225168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.225142 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.239383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.239357 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 29 22:24:51.239591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.239425 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-bkmtr\"" Oct 29 22:24:51.239591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.239464 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 29 22:24:51.239721 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.239500 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 29 22:24:51.251037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.251012 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5"] Oct 29 22:24:51.348273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.348243 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:51.361410 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.361365 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4f450ff-4b49-4058-b373-d66951b19315-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.361542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.361467 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544gx\" (UniqueName: \"kubernetes.io/projected/a4f450ff-4b49-4058-b373-d66951b19315-kube-api-access-544gx\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.361542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.361510 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f450ff-4b49-4058-b373-d66951b19315-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.361542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.361534 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a4f450ff-4b49-4058-b373-d66951b19315-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.462929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.462893 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4f450ff-4b49-4058-b373-d66951b19315-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.462929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.462932 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544gx\" (UniqueName: \"kubernetes.io/projected/a4f450ff-4b49-4058-b373-d66951b19315-kube-api-access-544gx\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.463117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.462963 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f450ff-4b49-4058-b373-d66951b19315-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.463117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.462986 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a4f450ff-4b49-4058-b373-d66951b19315-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.463690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.463659 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a4f450ff-4b49-4058-b373-d66951b19315-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.463814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.463726 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a4f450ff-4b49-4058-b373-d66951b19315-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.465308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.465291 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4f450ff-4b49-4058-b373-d66951b19315-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.479740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.479684 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544gx\" (UniqueName: \"kubernetes.io/projected/a4f450ff-4b49-4058-b373-d66951b19315-kube-api-access-544gx\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-rdht5\" (UID: \"a4f450ff-4b49-4058-b373-d66951b19315\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.480532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.480497 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:24:51.537649 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.537619 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:24:51.829949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:51.829843 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5"] Oct 29 22:24:51.832664 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:24:51.832633 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f450ff_4b49_4058_b373_d66951b19315.slice/crio-6cc3648dc0ef8ea6e0942dc4b589d13fb77c28682610d71670af289fc2ba832e WatchSource:0}: Error finding container 6cc3648dc0ef8ea6e0942dc4b589d13fb77c28682610d71670af289fc2ba832e: Status 404 returned error can't find the container with id 6cc3648dc0ef8ea6e0942dc4b589d13fb77c28682610d71670af289fc2ba832e Oct 29 22:24:52.816826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:52.816793 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" event={"ID":"a4f450ff-4b49-4058-b373-d66951b19315","Type":"ContainerStarted","Data":"6cc3648dc0ef8ea6e0942dc4b589d13fb77c28682610d71670af289fc2ba832e"} Oct 29 22:24:58.802447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:58.802412 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 22:24:58.867735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:58.867698 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-brs7d"] Oct 29 22:24:58.868253 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:58.868215 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-brs7d" podUID="99169dd1-cd7a-48bc-8b6b-4040ebecd14c" containerName="manager" containerID="cri-o://8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86" gracePeriod=30 Oct 29 22:24:58.870223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:58.870199 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:59.131139 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.131114 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:59.237370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.237335 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs\") pod \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " Oct 29 22:24:59.237576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.237419 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-webhook-cert\") pod \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " Oct 29 22:24:59.237576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.237454 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-kube-api-access-zq8gf\" (UniqueName: \"kubernetes.io/projected/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-kube-api-access-zq8gf\") pod \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\" (UID: \"99169dd1-cd7a-48bc-8b6b-4040ebecd14c\") " Oct 29 22:24:59.239987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.239921 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs" pod "99169dd1-cd7a-48bc-8b6b-4040ebecd14c" (UID: "99169dd1-cd7a-48bc-8b6b-4040ebecd14c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:24:59.240101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.240010 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-webhook-cert" pod "99169dd1-cd7a-48bc-8b6b-4040ebecd14c" (UID: "99169dd1-cd7a-48bc-8b6b-4040ebecd14c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:24:59.240101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.240072 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-kube-api-access-zq8gf" pod "99169dd1-cd7a-48bc-8b6b-4040ebecd14c" (UID: "99169dd1-cd7a-48bc-8b6b-4040ebecd14c"). InnerVolumeSpecName "kube-api-access-zq8gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:24:59.338320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.338281 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:24:59.338320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.338314 5610 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-webhook-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:24:59.338530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.338331 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8gf\" (UniqueName: \"kubernetes.io/projected/99169dd1-cd7a-48bc-8b6b-4040ebecd14c-kube-api-access-zq8gf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:24:59.855005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.854966 5610 generic.go:356] "Generic (PLEG): container finished" podID="99169dd1-cd7a-48bc-8b6b-4040ebecd14c" containerID="8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86" exitCode=0 Oct 29 22:24:59.855467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.855053 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-brs7d" Oct 29 22:24:59.855467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.855065 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-brs7d" event={"ID":"99169dd1-cd7a-48bc-8b6b-4040ebecd14c","Type":"ContainerDied","Data":"8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86"} Oct 29 22:24:59.855467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.855130 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-brs7d" event={"ID":"99169dd1-cd7a-48bc-8b6b-4040ebecd14c","Type":"ContainerDied","Data":"cd3673bbebc0d6badbfc0105caf7bae1ebb9771f81ac16cdd5c3fc34f7ec3709"} Oct 29 22:24:59.855467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.855160 5610 scope.go:122] "RemoveContainer" containerID="8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86" Oct 29 22:24:59.865185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.865161 5610 scope.go:122] "RemoveContainer" containerID="8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86" Oct 29 22:24:59.865458 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:24:59.865434 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86\": container with ID starting with 8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86 not found: ID does not exist" containerID="8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86" Oct 29 22:24:59.865555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.865467 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86"} err="failed to get container status \"8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86\": rpc error: code = NotFound desc = could not find container \"8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86\": container with ID starting with 8717dca926d211d2260ab70eae3765f96e21f7ce7b220450ffd85d844f170c86 not found: ID does not exist" Oct 29 22:24:59.899979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.899910 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-brs7d"] Oct 29 22:24:59.910258 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:24:59.910236 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-brs7d"] Oct 29 22:25:01.635055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:01.635022 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99169dd1-cd7a-48bc-8b6b-4040ebecd14c" path="/var/lib/kubelet/pods/99169dd1-cd7a-48bc-8b6b-4040ebecd14c/volumes" Oct 29 22:25:02.869405 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:02.869372 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" event={"ID":"a4f450ff-4b49-4058-b373-d66951b19315","Type":"ContainerStarted","Data":"2bbd409204e2c7b65128e8586fae2ae8d76112221ee6ad8a81ce0f9bedbdb150"} Oct 29 22:25:02.869839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:02.869482 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:25:02.903553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:02.903510 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" podStartSLOduration=1.8076352770000002 podStartE2EDuration="11.903498223s" totalImagesPullingTime="10.095862946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:02.901996067 +0000 UTC m=+565.383313559" watchObservedRunningTime="2026-10-29 22:25:02.903498223 +0000 UTC m=+565.384815715" Oct 29 22:25:13.709454 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:25:13.709375 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:25:13.879987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:13.879957 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" Oct 29 22:25:17.319881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:17.319846 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:17.471006 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:17.470972 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:18.796950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:18.796910 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:18.935776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:18.935746 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:18.994510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:18.994474 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vchhh"] Oct 29 22:25:18.994964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:18.994938 5610 cpu_manager.go:414] "containerMap: removing container" podUID="99169dd1-cd7a-48bc-8b6b-4040ebecd14c" containerName="manager" Oct 29 22:25:18.995015 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:18.994974 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99169dd1-cd7a-48bc-8b6b-4040ebecd14c" containerName="manager" Oct 29 22:25:18.995076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:18.995063 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="99169dd1-cd7a-48bc-8b6b-4040ebecd14c" containerName="manager" Oct 29 22:25:19.003733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.003710 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.019223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.019197 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vchhh"] Oct 29 22:25:19.048163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.048091 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 29 22:25:19.048339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.048197 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 29 22:25:19.048339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.048241 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dv786\"" Oct 29 22:25:19.106023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.105989 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-maas-api-tls\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.106202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.106043 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-metrics-certs\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.106202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.106122 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe896cf-231e-4c94-b414-ae90fb96c5dc-openshift-service-ca\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.106202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.106181 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pndzf\" (UniqueName: \"kubernetes.io/projected/ebe896cf-231e-4c94-b414-ae90fb96c5dc-kube-api-access-pndzf\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.207541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.207506 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-maas-api-tls\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.207722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.207558 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-metrics-certs\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.207722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.207589 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe896cf-231e-4c94-b414-ae90fb96c5dc-openshift-service-ca\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.207722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.207621 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pndzf\" (UniqueName: \"kubernetes.io/projected/ebe896cf-231e-4c94-b414-ae90fb96c5dc-kube-api-access-pndzf\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.208420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.208369 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe896cf-231e-4c94-b414-ae90fb96c5dc-openshift-service-ca\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.209977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.209955 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-maas-api-tls\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.210156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.210135 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-metrics-certs\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.224322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.224291 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pndzf\" (UniqueName: \"kubernetes.io/projected/ebe896cf-231e-4c94-b414-ae90fb96c5dc-kube-api-access-pndzf\") pod \"maas-api-5c48877c75-vchhh\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.317846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.317748 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:19.667315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.667129 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vchhh"] Oct 29 22:25:19.670168 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:25:19.670142 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe896cf_231e_4c94_b414_ae90fb96c5dc.slice/crio-301485068e0f2bf34815f894979033ef6edc01b4d6e2599fffcf28791021a1bd WatchSource:0}: Error finding container 301485068e0f2bf34815f894979033ef6edc01b4d6e2599fffcf28791021a1bd: Status 404 returned error can't find the container with id 301485068e0f2bf34815f894979033ef6edc01b4d6e2599fffcf28791021a1bd Oct 29 22:25:19.671569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.671547 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:25:19.945361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:19.945332 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" event={"ID":"ebe896cf-231e-4c94-b414-ae90fb96c5dc","Type":"ContainerStarted","Data":"301485068e0f2bf34815f894979033ef6edc01b4d6e2599fffcf28791021a1bd"} Oct 29 22:25:21.957953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:21.957912 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" event={"ID":"ebe896cf-231e-4c94-b414-ae90fb96c5dc","Type":"ContainerStarted","Data":"8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5"} Oct 29 22:25:21.958355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:21.957999 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:22.001401 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:22.001337 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" podStartSLOduration=1.981080153 podStartE2EDuration="4.001323644s" totalImagesPullingTime="2.020243491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:25:22.000256883 +0000 UTC m=+584.481574405" watchObservedRunningTime="2026-10-29 22:25:22.001323644 +0000 UTC m=+584.482641138" Oct 29 22:25:27.967811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:27.967779 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 22:25:35.516669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:25:35.516634 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:25:43.708934 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:25:43.708883 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:26:02.909227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:02.909195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:03.051851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:03.051795 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:13.708576 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:26:13.708532 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:26:21.727969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:21.727883 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:23.640079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:23.640041 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:24.996852 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:24.996816 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk"] Oct 29 22:26:25.001443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.001421 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.026640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.026614 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-59nr7\"" Oct 29 22:26:25.026954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.026937 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 29 22:26:25.027086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.027027 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 29 22:26:25.028742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.028721 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk"] Oct 29 22:26:25.030348 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.030329 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 29 22:26:25.056789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.056762 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.056899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.056805 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c7e838-5a9d-477c-994e-e02cff87ba8c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.056899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.056867 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhnd\" (UniqueName: \"kubernetes.io/projected/f7c7e838-5a9d-477c-994e-e02cff87ba8c-kube-api-access-wlhnd\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.056899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.056891 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.057020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.056929 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.057020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.056972 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.115603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.115569 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:25.157463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157427 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.157463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157467 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.157662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157506 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.157662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157543 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c7e838-5a9d-477c-994e-e02cff87ba8c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.157662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157579 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhnd\" (UniqueName: \"kubernetes.io/projected/f7c7e838-5a9d-477c-994e-e02cff87ba8c-kube-api-access-wlhnd\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.157662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157617 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.157931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157904 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.157995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157917 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.158038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.157990 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.159865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.159846 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c7e838-5a9d-477c-994e-e02cff87ba8c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.159957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.159940 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c7e838-5a9d-477c-994e-e02cff87ba8c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.175331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.175311 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhnd\" (UniqueName: \"kubernetes.io/projected/f7c7e838-5a9d-477c-994e-e02cff87ba8c-kube-api-access-wlhnd\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dwclk\" (UID: \"f7c7e838-5a9d-477c-994e-e02cff87ba8c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.316988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.316911 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:25.609286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:25.609250 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk"] Oct 29 22:26:25.612092 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:26:25.612059 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c7e838_5a9d_477c_994e_e02cff87ba8c.slice/crio-b148422df420a826b303017edc0c541103ac92294d39972137f37ed4a8bf2874 WatchSource:0}: Error finding container b148422df420a826b303017edc0c541103ac92294d39972137f37ed4a8bf2874: Status 404 returned error can't find the container with id b148422df420a826b303017edc0c541103ac92294d39972137f37ed4a8bf2874 Oct 29 22:26:26.240295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.240244 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" event={"ID":"f7c7e838-5a9d-477c-994e-e02cff87ba8c","Type":"ContainerStarted","Data":"b148422df420a826b303017edc0c541103ac92294d39972137f37ed4a8bf2874"} Oct 29 22:26:26.349441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.349406 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2"] Oct 29 22:26:26.354726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.353847 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.380306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.378260 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 29 22:26:26.380306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.379476 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2"] Oct 29 22:26:26.461116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.461080 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:26.474051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.474015 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.474217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.474114 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.474217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.474183 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.474217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.474212 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw8gb\" (UniqueName: \"kubernetes.io/projected/04592e8c-6304-4b3e-8f94-2629a2ff3162-kube-api-access-zw8gb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.474383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.474278 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.474383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.474297 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04592e8c-6304-4b3e-8f94-2629a2ff3162-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.574991 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.575043 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04592e8c-6304-4b3e-8f94-2629a2ff3162-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.575087 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.575108 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.575160 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.575219 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw8gb\" (UniqueName: \"kubernetes.io/projected/04592e8c-6304-4b3e-8f94-2629a2ff3162-kube-api-access-zw8gb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.575480 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.575609 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.575787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.575754 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.577751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.577725 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04592e8c-6304-4b3e-8f94-2629a2ff3162-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.578277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.578255 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04592e8c-6304-4b3e-8f94-2629a2ff3162-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.592509 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.592481 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw8gb\" (UniqueName: \"kubernetes.io/projected/04592e8c-6304-4b3e-8f94-2629a2ff3162-kube-api-access-zw8gb\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2\" (UID: \"04592e8c-6304-4b3e-8f94-2629a2ff3162\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.673456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.673363 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:26.979013 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:26.978887 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2"] Oct 29 22:26:26.981326 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:26:26.981298 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04592e8c_6304_4b3e_8f94_2629a2ff3162.slice/crio-84445b5b42fba2e36092fd38aad26ba07c6f741915b3c67c2e84be8b67fe2b7a WatchSource:0}: Error finding container 84445b5b42fba2e36092fd38aad26ba07c6f741915b3c67c2e84be8b67fe2b7a: Status 404 returned error can't find the container with id 84445b5b42fba2e36092fd38aad26ba07c6f741915b3c67c2e84be8b67fe2b7a Oct 29 22:26:27.245536 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:27.245510 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" event={"ID":"04592e8c-6304-4b3e-8f94-2629a2ff3162","Type":"ContainerStarted","Data":"84445b5b42fba2e36092fd38aad26ba07c6f741915b3c67c2e84be8b67fe2b7a"} Oct 29 22:26:27.247034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:27.247008 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" event={"ID":"f7c7e838-5a9d-477c-994e-e02cff87ba8c","Type":"ContainerStarted","Data":"d34cac1aac0a7e82807125e0af6484e0c72c3f3622a8b307e1a177555dbc73e4"} Oct 29 22:26:27.282238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:27.282189 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" podStartSLOduration=2.17089942 podStartE2EDuration="3.282175611s" totalImagesPullingTime="1.111276191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:27.280796405 +0000 UTC m=+649.762113898" watchObservedRunningTime="2026-10-29 22:26:27.282175611 +0000 UTC m=+649.763493101" Oct 29 22:26:27.317063 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:27.317034 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:28.252885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:28.252852 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" event={"ID":"04592e8c-6304-4b3e-8f94-2629a2ff3162","Type":"ContainerStarted","Data":"0a25d26046bb03c04f7abf52a661a9f9e61a1a97ad6b7067522b662daa810034"} Oct 29 22:26:28.291937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:28.291891 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" podStartSLOduration=2.075177413 podStartE2EDuration="2.291877161s" totalImagesPullingTime="216.699748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:28.291426204 +0000 UTC m=+650.772743694" watchObservedRunningTime="2026-10-29 22:26:28.291877161 +0000 UTC m=+650.773194653" Oct 29 22:26:28.674190 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:28.674115 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:29.597844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.597810 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j"] Oct 29 22:26:29.601202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.601174 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.613850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.613824 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 29 22:26:29.624843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.624820 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j"] Oct 29 22:26:29.701433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.701383 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.701609 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.701484 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fcae09-de89-49fc-98f1-3bdaf39751b3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.701680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.701618 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.701746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.701722 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.701803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.701758 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.702008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.701984 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hb25\" (UniqueName: \"kubernetes.io/projected/10fcae09-de89-49fc-98f1-3bdaf39751b3-kube-api-access-8hb25\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.768158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.768118 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:29.803031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803002 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fcae09-de89-49fc-98f1-3bdaf39751b3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.803184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803040 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.803184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803066 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.803299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803193 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.803299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803283 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hb25\" (UniqueName: \"kubernetes.io/projected/10fcae09-de89-49fc-98f1-3bdaf39751b3-kube-api-access-8hb25\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.803431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803333 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.803498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803448 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.803551 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803497 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.803551 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.803527 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.805461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.805435 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10fcae09-de89-49fc-98f1-3bdaf39751b3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.805621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.805602 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10fcae09-de89-49fc-98f1-3bdaf39751b3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.819998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.819975 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hb25\" (UniqueName: \"kubernetes.io/projected/10fcae09-de89-49fc-98f1-3bdaf39751b3-kube-api-access-8hb25\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j\" (UID: \"10fcae09-de89-49fc-98f1-3bdaf39751b3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:29.914935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:29.914866 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:30.204677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:30.204649 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j"] Oct 29 22:26:30.207206 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:26:30.207171 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fcae09_de89_49fc_98f1_3bdaf39751b3.slice/crio-c2700fbd9c13c4106cf24fe521413e68493cfe8bac4fe5d0f8a9b412a82dc681 WatchSource:0}: Error finding container c2700fbd9c13c4106cf24fe521413e68493cfe8bac4fe5d0f8a9b412a82dc681: Status 404 returned error can't find the container with id c2700fbd9c13c4106cf24fe521413e68493cfe8bac4fe5d0f8a9b412a82dc681 Oct 29 22:26:30.262462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:30.262431 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" event={"ID":"10fcae09-de89-49fc-98f1-3bdaf39751b3","Type":"ContainerStarted","Data":"c2700fbd9c13c4106cf24fe521413e68493cfe8bac4fe5d0f8a9b412a82dc681"} Oct 29 22:26:31.269192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:31.269159 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" event={"ID":"10fcae09-de89-49fc-98f1-3bdaf39751b3","Type":"ContainerStarted","Data":"b0de066333b3188284b3db380ed2304d63a36d048744326d89f72774f5ab84f6"} Oct 29 22:26:31.308087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:31.308035 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" podStartSLOduration=2.102233777 podStartE2EDuration="2.308019726s" totalImagesPullingTime="205.785949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:31.306759252 +0000 UTC m=+653.788076748" watchObservedRunningTime="2026-10-29 22:26:31.308019726 +0000 UTC m=+653.789337218" Oct 29 22:26:31.915416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:31.915365 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:34.108275 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:34.108232 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:35.317161 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:35.317124 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:35.331018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:35.330995 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:35.426442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:35.426408 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:35.651479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:35.651387 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:36.304568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:36.304537 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" Oct 29 22:26:36.673827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:36.673750 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:36.685825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:36.685801 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:37.310186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:37.310154 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" Oct 29 22:26:39.915370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:39.915327 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:39.933305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:39.933271 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:40.323850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:40.323823 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" Oct 29 22:26:43.708601 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:26:43.708570 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:26:48.858986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:48.858941 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:49.680444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.680412 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-2tckw"] Oct 29 22:26:49.685536 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.685514 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:49.707725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.707690 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5vzpj\"" Oct 29 22:26:49.707725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.707689 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 29 22:26:49.707952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.707766 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 29 22:26:49.709151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.709131 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-2tckw"] Oct 29 22:26:49.712776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.712754 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:49.786130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.786076 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67c1874-831d-4bee-aae7-0c8a85579ee7-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-2tckw\" (UID: \"e67c1874-831d-4bee-aae7-0c8a85579ee7\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:49.786287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.786216 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gtm\" (UniqueName: \"kubernetes.io/projected/e67c1874-831d-4bee-aae7-0c8a85579ee7-kube-api-access-26gtm\") pod \"ai-gateway-controller-55b4c89cbd-2tckw\" (UID: \"e67c1874-831d-4bee-aae7-0c8a85579ee7\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:49.852657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.852630 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:49.887054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.887029 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67c1874-831d-4bee-aae7-0c8a85579ee7-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-2tckw\" (UID: \"e67c1874-831d-4bee-aae7-0c8a85579ee7\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:49.887388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.887063 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gtm\" (UniqueName: \"kubernetes.io/projected/e67c1874-831d-4bee-aae7-0c8a85579ee7-kube-api-access-26gtm\") pod \"ai-gateway-controller-55b4c89cbd-2tckw\" (UID: \"e67c1874-831d-4bee-aae7-0c8a85579ee7\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:49.889431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.889407 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67c1874-831d-4bee-aae7-0c8a85579ee7-metrics-certs\") pod \"ai-gateway-controller-55b4c89cbd-2tckw\" (UID: \"e67c1874-831d-4bee-aae7-0c8a85579ee7\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:49.903444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.903420 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gtm\" (UniqueName: \"kubernetes.io/projected/e67c1874-831d-4bee-aae7-0c8a85579ee7-kube-api-access-26gtm\") pod \"ai-gateway-controller-55b4c89cbd-2tckw\" (UID: \"e67c1874-831d-4bee-aae7-0c8a85579ee7\") " pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:49.998403 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:49.998370 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:50.295493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:50.295409 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-2tckw"] Oct 29 22:26:50.298737 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:26:50.298709 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67c1874_831d_4bee_aae7_0c8a85579ee7.slice/crio-21f6aea8e90a88321d09efe0fdc3e4c2c95e2d19a616084a38e52cec492f68ba WatchSource:0}: Error finding container 21f6aea8e90a88321d09efe0fdc3e4c2c95e2d19a616084a38e52cec492f68ba: Status 404 returned error can't find the container with id 21f6aea8e90a88321d09efe0fdc3e4c2c95e2d19a616084a38e52cec492f68ba Oct 29 22:26:50.358222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:50.358190 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" event={"ID":"e67c1874-831d-4bee-aae7-0c8a85579ee7","Type":"ContainerStarted","Data":"21f6aea8e90a88321d09efe0fdc3e4c2c95e2d19a616084a38e52cec492f68ba"} Oct 29 22:26:52.486714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:52.486670 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:52.619177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:52.619143 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:53.375657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:53.375622 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" event={"ID":"e67c1874-831d-4bee-aae7-0c8a85579ee7","Type":"ContainerStarted","Data":"e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120"} Oct 29 22:26:53.375898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:53.375880 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:26:53.410287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:53.410233 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" podStartSLOduration=2.082721549 podStartE2EDuration="4.410216306s" totalImagesPullingTime="2.327494757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:53.40948541 +0000 UTC m=+675.890802902" watchObservedRunningTime="2026-10-29 22:26:53.410216306 +0000 UTC m=+675.891533800" Oct 29 22:26:54.789647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:54.789607 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:54.854299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:54.854266 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-zt6f2"] Oct 29 22:26:54.857799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:54.857779 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:54.883444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:54.883419 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rb8ck\"" Oct 29 22:26:54.883444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:54.883416 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 29 22:26:54.884701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:54.884679 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-zt6f2"] Oct 29 22:26:54.929244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:54.929211 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:26:55.035298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:55.035262 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0554501a-13e6-4813-b5d0-b5cb2b14f4c5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-zt6f2\" (UID: \"0554501a-13e6-4813-b5d0-b5cb2b14f4c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:55.035504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:55.035463 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjdk\" (UniqueName: \"kubernetes.io/projected/0554501a-13e6-4813-b5d0-b5cb2b14f4c5-kube-api-access-5wjdk\") pod \"payload-processing-5f78687d5b-zt6f2\" (UID: \"0554501a-13e6-4813-b5d0-b5cb2b14f4c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:55.136740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:55.136661 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjdk\" (UniqueName: \"kubernetes.io/projected/0554501a-13e6-4813-b5d0-b5cb2b14f4c5-kube-api-access-5wjdk\") pod \"payload-processing-5f78687d5b-zt6f2\" (UID: \"0554501a-13e6-4813-b5d0-b5cb2b14f4c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:55.136740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:55.136719 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0554501a-13e6-4813-b5d0-b5cb2b14f4c5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-zt6f2\" (UID: \"0554501a-13e6-4813-b5d0-b5cb2b14f4c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:55.137356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:55.137337 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0554501a-13e6-4813-b5d0-b5cb2b14f4c5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-zt6f2\" (UID: \"0554501a-13e6-4813-b5d0-b5cb2b14f4c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:55.153293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:55.153262 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjdk\" (UniqueName: \"kubernetes.io/projected/0554501a-13e6-4813-b5d0-b5cb2b14f4c5-kube-api-access-5wjdk\") pod \"payload-processing-5f78687d5b-zt6f2\" (UID: \"0554501a-13e6-4813-b5d0-b5cb2b14f4c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:55.169927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:55.169898 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:55.463072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:55.463039 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-zt6f2"] Oct 29 22:26:55.467077 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:26:55.467042 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0554501a_13e6_4813_b5d0_b5cb2b14f4c5.slice/crio-cc602446baf164c6a0311cd43490a39625bf13bf75959c6fc7b2c4bec30bc13b WatchSource:0}: Error finding container cc602446baf164c6a0311cd43490a39625bf13bf75959c6fc7b2c4bec30bc13b: Status 404 returned error can't find the container with id cc602446baf164c6a0311cd43490a39625bf13bf75959c6fc7b2c4bec30bc13b Oct 29 22:26:56.390154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:56.390118 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" event={"ID":"0554501a-13e6-4813-b5d0-b5cb2b14f4c5","Type":"ContainerStarted","Data":"cc602446baf164c6a0311cd43490a39625bf13bf75959c6fc7b2c4bec30bc13b"} Oct 29 22:26:57.123156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.122811 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9"] Oct 29 22:26:57.143651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.143620 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.145303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.145274 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j"] Oct 29 22:26:57.149977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.149953 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.154840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.154045 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j"] Oct 29 22:26:57.164481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.164454 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9"] Oct 29 22:26:57.245192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.245165 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lf5dq\"" Oct 29 22:26:57.258491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258446 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.258623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258496 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.258623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258575 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.258623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258600 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.258623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258621 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.258779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258744 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.258779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258770 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.258885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258850 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.258927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258884 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfhs\" (UniqueName: \"kubernetes.io/projected/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-kube-api-access-vpfhs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.258927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258919 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.259002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.258936 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.259039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.259012 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.259071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.259041 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.259071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.259060 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.259129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.259087 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.259129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.259125 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.259205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.259142 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.259205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.259185 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgf5s\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-kube-api-access-pgf5s\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.359878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.359847 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.359884 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.359905 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.359924 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.359947 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.359989 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360018 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360062 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360089 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfhs\" (UniqueName: \"kubernetes.io/projected/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-kube-api-access-vpfhs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360127 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360152 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360208 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360231 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360258 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360281 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360303 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360313 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360348 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360517 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360317 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.360702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360640 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.360702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360668 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgf5s\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-kube-api-access-pgf5s\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.361102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.360776 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.361102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.361041 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.361209 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.361158 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.361262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.361215 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.361319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.361300 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.361535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.361508 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.362490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.362471 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.363018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.362995 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.363155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.363140 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.363669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.363652 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.376090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.376035 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.376194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.376109 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.376429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.376384 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfhs\" (UniqueName: \"kubernetes.io/projected/a1d48ad6-b38a-4ed8-b13a-44b591cd02d5-kube-api-access-vpfhs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j\" (UID: \"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.376897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.376872 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgf5s\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-kube-api-access-pgf5s\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.397589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.397562 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" event={"ID":"0554501a-13e6-4813-b5d0-b5cb2b14f4c5","Type":"ContainerStarted","Data":"d99d3855c5e3b93b67365d8de2b625b387bd82b90d715f4b9849ce462b31af7c"} Oct 29 22:26:57.397927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.397718 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:26:57.431622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.431571 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" podStartSLOduration=1.63151162 podStartE2EDuration="3.431557789s" totalImagesPullingTime="1.800046169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:26:57.43037428 +0000 UTC m=+679.911691774" watchObservedRunningTime="2026-10-29 22:26:57.431557789 +0000 UTC m=+679.912875297" Oct 29 22:26:57.467885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.467854 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:26:57.476996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.476972 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:26:57.788245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.788221 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9"] Oct 29 22:26:57.790525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:57.790501 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j"] Oct 29 22:26:57.790665 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:26:57.790626 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6797f453_aa2d_4d46_8d2d_1a3c4907db03.slice/crio-368bf085ba04da6b2643cb8d7539545eef8021be21415d287c96b8ff7ae5cc29 WatchSource:0}: Error finding container 368bf085ba04da6b2643cb8d7539545eef8021be21415d287c96b8ff7ae5cc29: Status 404 returned error can't find the container with id 368bf085ba04da6b2643cb8d7539545eef8021be21415d287c96b8ff7ae5cc29 Oct 29 22:26:57.801172 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:26:57.801142 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d48ad6_b38a_4ed8_b13a_44b591cd02d5.slice/crio-bd5cd3160cab77992575a771cb144756e4bc74afa3b04cd9f24bdc705ff6584e WatchSource:0}: Error finding container bd5cd3160cab77992575a771cb144756e4bc74afa3b04cd9f24bdc705ff6584e: Status 404 returned error can't find the container with id bd5cd3160cab77992575a771cb144756e4bc74afa3b04cd9f24bdc705ff6584e Oct 29 22:26:58.406681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:58.406645 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" event={"ID":"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5","Type":"ContainerStarted","Data":"bd5cd3160cab77992575a771cb144756e4bc74afa3b04cd9f24bdc705ff6584e"} Oct 29 22:26:58.408442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:26:58.408365 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" event={"ID":"6797f453-aa2d-4d46-8d2d-1a3c4907db03","Type":"ContainerStarted","Data":"368bf085ba04da6b2643cb8d7539545eef8021be21415d287c96b8ff7ae5cc29"} Oct 29 22:27:00.623843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:00.623805 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:00.624152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:00.623894 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:00.624152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:00.623934 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:00.631550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:00.631519 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:00.631645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:00.631592 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:00.631705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:00.631640 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:01.426163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.426126 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" event={"ID":"a1d48ad6-b38a-4ed8-b13a-44b591cd02d5","Type":"ContainerStarted","Data":"47e2c68e38ca3bb93d4a5615c2a1d439548af6c5131cdf7cb3387bad77614681"} Oct 29 22:27:01.427983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.427956 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" event={"ID":"6797f453-aa2d-4d46-8d2d-1a3c4907db03","Type":"ContainerStarted","Data":"3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9"} Oct 29 22:27:01.481412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.480850 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podStartSLOduration=1.652968816 podStartE2EDuration="4.48083165s" totalImagesPullingTime="2.827862834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:01.479856942 +0000 UTC m=+683.961174448" watchObservedRunningTime="2026-10-29 22:27:01.48083165 +0000 UTC m=+683.962149144" Oct 29 22:27:01.482417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.482272 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:27:01.485424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.482841 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:27:01.506790 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.505538 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:27:01.506790 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.505607 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 29 22:27:01.507191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.507168 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:27:01.507319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.507298 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 29 22:27:01.531029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:01.530944 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podStartSLOduration=1.700867745 podStartE2EDuration="4.530928591s" totalImagesPullingTime="2.830060846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:01.528918702 +0000 UTC m=+684.010236196" watchObservedRunningTime="2026-10-29 22:27:01.530928591 +0000 UTC m=+684.012246083" Oct 29 22:27:02.479418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:02.477495 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:27:02.479418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:02.477555 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 29 22:27:02.479418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:02.478476 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:27:02.479418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:02.478518 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 29 22:27:03.478414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:03.476802 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:27:03.478414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:03.476861 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 29 22:27:03.478414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:03.477686 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:27:03.478414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:03.477726 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 29 22:27:04.404423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:04.392690 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:27:04.476417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:04.475490 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:27:04.476417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:04.475555 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 29 22:27:04.481418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:04.477712 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:27:04.481418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:04.477759 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 29 22:27:05.475416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:05.471751 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:27:05.475416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:05.471811 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 29 22:27:05.481418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:05.480712 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:27:05.481418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:05.480768 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 29 22:27:06.037831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:06.037804 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:06.403482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:06.402793 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:06.476419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:06.468295 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:27:06.476419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:06.468351 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 29 22:27:06.505418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:06.490237 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:27:06.505418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:06.490294 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 29 22:27:07.469112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:07.469075 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 29 22:27:07.469269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:07.469142 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 29 22:27:07.477447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:07.477412 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 29 22:27:07.477784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:07.477472 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 29 22:27:08.412998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:08.412963 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" Oct 29 22:27:09.097260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:09.097223 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:27:09.097711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:09.097632 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:27:09.098551 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:09.098522 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" Oct 29 22:27:09.100725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:09.100702 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:27:09.101131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:09.101110 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:27:09.101902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:09.101882 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:27:13.708554 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:27:13.708517 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:27:19.234204 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.234173 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-2tckw"] Oct 29 22:27:19.234725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.234383 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" podUID="e67c1874-831d-4bee-aae7-0c8a85579ee7" containerName="manager" containerID="cri-o://e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120" gracePeriod=30 Oct 29 22:27:19.486904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.486845 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:27:19.519043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.518860 5610 generic.go:356] "Generic (PLEG): container finished" podID="e67c1874-831d-4bee-aae7-0c8a85579ee7" containerID="e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120" exitCode=0 Oct 29 22:27:19.519043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.518922 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" Oct 29 22:27:19.519043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.518954 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" event={"ID":"e67c1874-831d-4bee-aae7-0c8a85579ee7","Type":"ContainerDied","Data":"e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120"} Oct 29 22:27:19.519043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.519034 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55b4c89cbd-2tckw" event={"ID":"e67c1874-831d-4bee-aae7-0c8a85579ee7","Type":"ContainerDied","Data":"21f6aea8e90a88321d09efe0fdc3e4c2c95e2d19a616084a38e52cec492f68ba"} Oct 29 22:27:19.519539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.519079 5610 scope.go:122] "RemoveContainer" containerID="e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120" Oct 29 22:27:19.529837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.529813 5610 scope.go:122] "RemoveContainer" containerID="e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120" Oct 29 22:27:19.530176 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:27:19.530151 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120\": container with ID starting with e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120 not found: ID does not exist" containerID="e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120" Oct 29 22:27:19.530227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.530190 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120"} err="failed to get container status \"e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120\": rpc error: code = NotFound desc = could not find container \"e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120\": container with ID starting with e38c1db9f069b19468e0580415a70173a18c16b966c7e26f21fe759c5db3a120 not found: ID does not exist" Oct 29 22:27:19.592461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.592428 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e67c1874-831d-4bee-aae7-0c8a85579ee7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67c1874-831d-4bee-aae7-0c8a85579ee7-metrics-certs\") pod \"e67c1874-831d-4bee-aae7-0c8a85579ee7\" (UID: \"e67c1874-831d-4bee-aae7-0c8a85579ee7\") " Oct 29 22:27:19.592723 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.592481 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67c1874-831d-4bee-aae7-0c8a85579ee7-kube-api-access-26gtm\" (UniqueName: \"kubernetes.io/projected/e67c1874-831d-4bee-aae7-0c8a85579ee7-kube-api-access-26gtm\") pod \"e67c1874-831d-4bee-aae7-0c8a85579ee7\" (UID: \"e67c1874-831d-4bee-aae7-0c8a85579ee7\") " Oct 29 22:27:19.594443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.594411 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67c1874-831d-4bee-aae7-0c8a85579ee7-metrics-certs" pod "e67c1874-831d-4bee-aae7-0c8a85579ee7" (UID: "e67c1874-831d-4bee-aae7-0c8a85579ee7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:27:19.594570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.594446 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67c1874-831d-4bee-aae7-0c8a85579ee7-kube-api-access-26gtm" pod "e67c1874-831d-4bee-aae7-0c8a85579ee7" (UID: "e67c1874-831d-4bee-aae7-0c8a85579ee7"). InnerVolumeSpecName "kube-api-access-26gtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:27:19.693872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.693838 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e67c1874-831d-4bee-aae7-0c8a85579ee7-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:27:19.693872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.693867 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26gtm\" (UniqueName: \"kubernetes.io/projected/e67c1874-831d-4bee-aae7-0c8a85579ee7-kube-api-access-26gtm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:27:19.853322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.853248 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-2tckw"] Oct 29 22:27:19.863147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:19.863119 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-55b4c89cbd-2tckw"] Oct 29 22:27:20.897342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:20.897307 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:21.464305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:21.464271 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:21.633944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:21.633903 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67c1874-831d-4bee-aae7-0c8a85579ee7" path="/var/lib/kubelet/pods/e67c1874-831d-4bee-aae7-0c8a85579ee7/volumes" Oct 29 22:27:22.997230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:22.997195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:23.151867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:23.151832 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:24.590697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:24.590667 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:24.730345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:24.730314 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:27.197632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.197592 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr"] Oct 29 22:27:27.198178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.198158 5610 cpu_manager.go:414] "containerMap: removing container" podUID="e67c1874-831d-4bee-aae7-0c8a85579ee7" containerName="manager" Oct 29 22:27:27.198257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.198181 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67c1874-831d-4bee-aae7-0c8a85579ee7" containerName="manager" Oct 29 22:27:27.198367 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.198321 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67c1874-831d-4bee-aae7-0c8a85579ee7" containerName="manager" Oct 29 22:27:27.201714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.201692 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.212083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.212061 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7"] Oct 29 22:27:27.215678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.215660 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.227712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.227692 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr"] Oct 29 22:27:27.237252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.237224 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7"] Oct 29 22:27:27.365363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365326 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zs9h\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-kube-api-access-5zs9h\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.365363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365365 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.365608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365386 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.365608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365506 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.365608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365536 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995527ab-3722-4cbb-858c-137621448ca3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.365608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365558 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmsj\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-kube-api-access-vlmsj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.365608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365598 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.365788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365680 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.365788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365713 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.365788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365733 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.365788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365769 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.365920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365807 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.365920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365833 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.365920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365849 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.365920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365862 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995527ab-3722-4cbb-858c-137621448ca3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.365920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365881 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.366095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.365999 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.366095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.366022 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467125 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467159 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467175 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995527ab-3722-4cbb-858c-137621448ca3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.467507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467229 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.467507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467289 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.467507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467318 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467544 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467587 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zs9h\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-kube-api-access-5zs9h\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.467667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467613 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467612 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467633 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467671 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.467993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467698 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995527ab-3722-4cbb-858c-137621448ca3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.467993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467870 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmsj\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-kube-api-access-vlmsj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467923 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.467993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.467967 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.468244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468013 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.468244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468039 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.468244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468067 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.468244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468138 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995527ab-3722-4cbb-858c-137621448ca3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.468490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468346 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.468490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468416 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.468752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468727 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.468814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468744 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.468814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468011 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.469010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.468988 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.469248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.469219 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.469296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.469247 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.470034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.470009 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.470159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.470142 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.470262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.470244 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995527ab-3722-4cbb-858c-137621448ca3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.470512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.470497 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.484290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.484261 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmsj\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-kube-api-access-vlmsj\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.484411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.484305 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.484700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.484679 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.484948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.484930 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zs9h\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-kube-api-access-5zs9h\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.515687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.515662 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:27.527809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.527785 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:27.838483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.838451 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr"] Oct 29 22:27:27.838593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.838516 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7"] Oct 29 22:27:27.839283 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:27.839258 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d34e19_375c_4f1f_a8ef_5df5966e1776.slice/crio-55895841d77ef002b73d9718530effa388a3b1f3863e7727b2c0fdbe4e0516eb WatchSource:0}: Error finding container 55895841d77ef002b73d9718530effa388a3b1f3863e7727b2c0fdbe4e0516eb: Status 404 returned error can't find the container with id 55895841d77ef002b73d9718530effa388a3b1f3863e7727b2c0fdbe4e0516eb Oct 29 22:27:27.839876 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:27.839828 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995527ab_3722_4cbb_858c_137621448ca3.slice/crio-7e42f306f463433d4817cacf942b717b4845a9762aeffc0c16109171ff874e46 WatchSource:0}: Error finding container 7e42f306f463433d4817cacf942b717b4845a9762aeffc0c16109171ff874e46: Status 404 returned error can't find the container with id 7e42f306f463433d4817cacf942b717b4845a9762aeffc0c16109171ff874e46 Oct 29 22:27:27.841564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.841451 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:27.841564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.841515 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:27.841564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.841544 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:27.842020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.841997 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:27.842067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.842049 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:27.842102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:27.842078 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:28.611722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:28.611682 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" event={"ID":"995527ab-3722-4cbb-858c-137621448ca3","Type":"ContainerStarted","Data":"fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2"} Oct 29 22:27:28.612199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:28.611762 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" event={"ID":"995527ab-3722-4cbb-858c-137621448ca3","Type":"ContainerStarted","Data":"7e42f306f463433d4817cacf942b717b4845a9762aeffc0c16109171ff874e46"} Oct 29 22:27:28.613872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:28.613845 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" event={"ID":"a0d34e19-375c-4f1f-a8ef-5df5966e1776","Type":"ContainerStarted","Data":"3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31"} Oct 29 22:27:28.613978 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:28.613896 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" event={"ID":"a0d34e19-375c-4f1f-a8ef-5df5966e1776","Type":"ContainerStarted","Data":"55895841d77ef002b73d9718530effa388a3b1f3863e7727b2c0fdbe4e0516eb"} Oct 29 22:27:28.687416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:28.684696 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podStartSLOduration=1.684674887 podStartE2EDuration="1.684674887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:28.657836341 +0000 UTC m=+711.139153835" watchObservedRunningTime="2026-10-29 22:27:28.684674887 +0000 UTC m=+711.165992381" Oct 29 22:27:28.711306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:28.711203 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podStartSLOduration=1.7111823099999999 podStartE2EDuration="1.71118231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:28.709012245 +0000 UTC m=+711.190329742" watchObservedRunningTime="2026-10-29 22:27:28.71118231 +0000 UTC m=+711.192499804" Oct 29 22:27:29.516021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:29.515990 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:29.517487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:29.517461 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:27:29.517615 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:29.517506 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 29 22:27:29.528646 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:29.528631 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:29.529582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:29.529558 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 22:27:29.529687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:29.529595 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 22:27:30.516653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:30.516612 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:27:30.517053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:30.516681 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 29 22:27:30.528411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:30.528369 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 22:27:30.528561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:30.528435 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 22:27:31.516341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:31.516299 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:27:31.516533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:31.516360 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 29 22:27:31.528676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:31.528643 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 22:27:31.529019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:31.528690 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 22:27:32.516563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:32.516525 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:27:32.516729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:32.516583 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 29 22:27:32.528497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:32.528475 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 22:27:32.528566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:32.528518 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 22:27:33.516827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:33.516790 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:27:33.517228 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:33.516850 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 29 22:27:33.528915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:33.528886 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 22:27:33.529015 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:33.528929 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 22:27:34.516966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:34.516936 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 29 22:27:34.517359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:34.516996 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 29 22:27:34.528256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:34.528228 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 29 22:27:34.528377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:34.528267 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 29 22:27:35.760258 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:35.760202 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:36.123463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:36.123364 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:36.124607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:36.124584 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:36.125347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:36.125329 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:27:36.194002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:36.193970 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:36.194856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:36.194668 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:36.195485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:36.195465 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:27:43.708701 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:27:43.708666 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:27:47.022751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.022713 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:47.163283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.163248 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:47.552908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.552870 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:47.721299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.721257 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47"] Oct 29 22:27:47.728766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.728730 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:47.732881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.732858 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.738088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.735345 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47"] Oct 29 22:27:47.782205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.782175 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-40cf9a-openshift-default-dockercfg-8bd6l\"" Oct 29 22:27:47.789590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.789562 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.789729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.789601 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.789729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.789674 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.789729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.789699 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca4a254-88f7-40a2-876c-6dc36cfd9259-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.789871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.789729 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.789871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.789804 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qtt\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-kube-api-access-n8qtt\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.789976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.789923 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.790024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.789990 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.790088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.790061 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.873116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.873035 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:47.891238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891203 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891239 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891269 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891291 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca4a254-88f7-40a2-876c-6dc36cfd9259-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891320 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891352 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qtt\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-kube-api-access-n8qtt\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891419 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891484 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891507 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891672 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891857 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.891935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891878 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.892428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.891940 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.892873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.892850 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca4a254-88f7-40a2-876c-6dc36cfd9259-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.894626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.894598 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.894778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.894758 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.916703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.916471 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qtt\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-kube-api-access-n8qtt\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:47.920001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:47.919974 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:48.021751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.021718 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh"] Oct 29 22:27:48.027953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.027918 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.040578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.040556 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-879596-openshift-default-dockercfg-gkrg6\"" Oct 29 22:27:48.047666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.047642 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh"] Oct 29 22:27:48.053966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.053940 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:48.072340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.072310 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:48.094495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094459 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.094495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094500 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.094707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094528 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573f8be8-460f-4353-b9ee-e85751503f91-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.094707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094615 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.094707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094680 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.094840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094718 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573f8be8-460f-4353-b9ee-e85751503f91-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.094840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094832 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.094946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094863 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.095023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.094944 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fq4\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-kube-api-access-l2fq4\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.196468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.196434 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.196468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.196474 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.196719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.196494 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573f8be8-460f-4353-b9ee-e85751503f91-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.196719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.196593 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.196719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.196623 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.196929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.196898 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.197003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.196940 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573f8be8-460f-4353-b9ee-e85751503f91-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.197003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.196952 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.197123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.197080 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.197177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.197131 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.197234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.197202 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fq4\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-kube-api-access-l2fq4\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.197283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.197234 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573f8be8-460f-4353-b9ee-e85751503f91-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.197440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.197419 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.197581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.197562 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.198924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.198901 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.199453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.199437 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573f8be8-460f-4353-b9ee-e85751503f91-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.216298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.216274 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.216988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.216958 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fq4\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-kube-api-access-l2fq4\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.218869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.218848 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:48.288197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.288162 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr"] Oct 29 22:27:48.294065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.294039 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.307969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.307947 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-95672c-openshift-default-dockercfg-hp48t\"" Oct 29 22:27:48.321905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.321876 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr"] Oct 29 22:27:48.342144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.342118 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:48.361652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.361622 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47"] Oct 29 22:27:48.363164 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:48.363140 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca4a254_88f7_40a2_876c_6dc36cfd9259.slice/crio-c5820f8ee3a3693437abcacb5ccb8ae380367a5dbf8db5ea384f65d6e5b8d3d1 WatchSource:0}: Error finding container c5820f8ee3a3693437abcacb5ccb8ae380367a5dbf8db5ea384f65d6e5b8d3d1: Status 404 returned error can't find the container with id c5820f8ee3a3693437abcacb5ccb8ae380367a5dbf8db5ea384f65d6e5b8d3d1 Oct 29 22:27:48.366046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.366013 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:48.366140 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.366097 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:48.366194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.366143 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:48.368002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.367976 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399722 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399754 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399778 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399840 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399856 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399877 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7cx\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-kube-api-access-gs7cx\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399922 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399949 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.401189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.399979 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.500950 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.500996 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501027 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7cx\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-kube-api-access-gs7cx\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501089 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501127 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501170 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501205 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501236 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501266 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501453 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501491 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.501888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.501773 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.502093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.502073 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.503792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.503764 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.504658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.504635 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.504864 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.504842 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.507439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.507380 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:48.518586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.518554 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.518586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.518557 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7cx\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-kube-api-access-gs7cx\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.626422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.614584 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:48.643377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.643213 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:48.694283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.694147 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh"] Oct 29 22:27:48.707951 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:48.707905 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573f8be8_460f_4353_b9ee_e85751503f91.slice/crio-000ac277d8e8fb33b4d9b05bf28a93f319d82789dac27586439b0322f59ca990 WatchSource:0}: Error finding container 000ac277d8e8fb33b4d9b05bf28a93f319d82789dac27586439b0322f59ca990: Status 404 returned error can't find the container with id 000ac277d8e8fb33b4d9b05bf28a93f319d82789dac27586439b0322f59ca990 Oct 29 22:27:48.711345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.711313 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:48.711449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.711384 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:48.711449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.711431 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:48.770045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.760704 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" event={"ID":"573f8be8-460f-4353-b9ee-e85751503f91","Type":"ContainerStarted","Data":"000ac277d8e8fb33b4d9b05bf28a93f319d82789dac27586439b0322f59ca990"} Oct 29 22:27:48.770045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.763524 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" event={"ID":"eca4a254-88f7-40a2-876c-6dc36cfd9259","Type":"ContainerStarted","Data":"e34ea2dd206d860bd88724660128aadffa7eb4e64d4da40194e6e13b33de9f63"} Oct 29 22:27:48.770045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.763576 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" event={"ID":"eca4a254-88f7-40a2-876c-6dc36cfd9259","Type":"ContainerStarted","Data":"c5820f8ee3a3693437abcacb5ccb8ae380367a5dbf8db5ea384f65d6e5b8d3d1"} Oct 29 22:27:48.806821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.806743 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podStartSLOduration=1.806724312 podStartE2EDuration="1.806724312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:48.803967363 +0000 UTC m=+731.285284858" watchObservedRunningTime="2026-10-29 22:27:48.806724312 +0000 UTC m=+731.288041806" Oct 29 22:27:48.995055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.995026 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr"] Oct 29 22:27:48.996858 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:48.996829 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295069a5_7550_4e03_b1aa_f6c7ae3aa310.slice/crio-7e994546a8fa9c5093f670ab08932c81a23b4468c0fbd434419c5449e123e552 WatchSource:0}: Error finding container 7e994546a8fa9c5093f670ab08932c81a23b4468c0fbd434419c5449e123e552: Status 404 returned error can't find the container with id 7e994546a8fa9c5093f670ab08932c81a23b4468c0fbd434419c5449e123e552 Oct 29 22:27:48.999498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.999457 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:48.999577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.999542 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:48.999613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:48.999586 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:27:49.054891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.054865 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:49.060565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.060535 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:49.769734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.769692 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" event={"ID":"295069a5-7550-4e03-b1aa-f6c7ae3aa310","Type":"ContainerStarted","Data":"43a669058423f61f015957066ca288335db15e39653dec0a834d736fba87aa69"} Oct 29 22:27:49.769938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.769757 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" event={"ID":"295069a5-7550-4e03-b1aa-f6c7ae3aa310","Type":"ContainerStarted","Data":"7e994546a8fa9c5093f670ab08932c81a23b4468c0fbd434419c5449e123e552"} Oct 29 22:27:49.771294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.771268 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" event={"ID":"573f8be8-460f-4353-b9ee-e85751503f91","Type":"ContainerStarted","Data":"0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1"} Oct 29 22:27:49.771454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.771434 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:49.772522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.772504 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:27:49.818197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.818105 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podStartSLOduration=1.818079724 podStartE2EDuration="1.818079724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:49.81757205 +0000 UTC m=+732.298889543" watchObservedRunningTime="2026-10-29 22:27:49.818079724 +0000 UTC m=+732.299397216" Oct 29 22:27:49.889581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:49.889530 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podStartSLOduration=2.889510798 podStartE2EDuration="2.889510798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:49.887147255 +0000 UTC m=+732.368464772" watchObservedRunningTime="2026-10-29 22:27:49.889510798 +0000 UTC m=+732.370828292" Oct 29 22:27:50.342714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.342682 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:50.347712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.347689 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:50.616200 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.616113 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:50.621003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.620977 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:50.777281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.777246 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:50.777492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.777324 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:50.777922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.777900 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:27:50.778321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.778303 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:27:50.926714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:50.926588 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:51.075061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:51.075017 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:51.231001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:51.230958 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:51.453685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:51.453642 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:51.608269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:51.608187 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:51.857421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:51.857353 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:52.003739 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.003709 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:52.152106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.152063 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:52.643824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.643796 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:52.777198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.777168 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw"] Oct 29 22:27:52.781087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.781065 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:52.801957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.801930 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw"] Oct 29 22:27:52.860416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.856874 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4a0287-dockercfg-m6hjg\"" Oct 29 22:27:52.860416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.856972 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4a0287\"" Oct 29 22:27:52.860416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.857757 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:52.878621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.878584 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4751898-78dc-401d-834b-d5bfb1dca7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw\" (UID: \"d4751898-78dc-401d-834b-d5bfb1dca7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:52.878844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.878825 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9vh\" (UniqueName: \"kubernetes.io/projected/d4751898-78dc-401d-834b-d5bfb1dca7ff-kube-api-access-mt9vh\") pod \"payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw\" (UID: \"d4751898-78dc-401d-834b-d5bfb1dca7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:52.980067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.980031 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9vh\" (UniqueName: \"kubernetes.io/projected/d4751898-78dc-401d-834b-d5bfb1dca7ff-kube-api-access-mt9vh\") pod \"payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw\" (UID: \"d4751898-78dc-401d-834b-d5bfb1dca7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:52.980264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.980141 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4751898-78dc-401d-834b-d5bfb1dca7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw\" (UID: \"d4751898-78dc-401d-834b-d5bfb1dca7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:52.980871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.980849 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4751898-78dc-401d-834b-d5bfb1dca7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw\" (UID: \"d4751898-78dc-401d-834b-d5bfb1dca7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:52.996435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:52.996413 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9vh\" (UniqueName: \"kubernetes.io/projected/d4751898-78dc-401d-834b-d5bfb1dca7ff-kube-api-access-mt9vh\") pod \"payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw\" (UID: \"d4751898-78dc-401d-834b-d5bfb1dca7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:53.096460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.096421 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:53.147778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.147744 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:53.295335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.295249 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:53.395066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.395044 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw"] Oct 29 22:27:53.397825 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:53.397794 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4751898_78dc_401d_834b_d5bfb1dca7ff.slice/crio-72a43e9d29f8bdc9d450c4031bd5b40f9f3c9b87137187da3b6f968e8fc891ae WatchSource:0}: Error finding container 72a43e9d29f8bdc9d450c4031bd5b40f9f3c9b87137187da3b6f968e8fc891ae: Status 404 returned error can't find the container with id 72a43e9d29f8bdc9d450c4031bd5b40f9f3c9b87137187da3b6f968e8fc891ae Oct 29 22:27:53.495067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.495032 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:53.795214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.795166 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" event={"ID":"d4751898-78dc-401d-834b-d5bfb1dca7ff","Type":"ContainerStarted","Data":"bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979"} Oct 29 22:27:53.795623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.795234 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" event={"ID":"d4751898-78dc-401d-834b-d5bfb1dca7ff","Type":"ContainerStarted","Data":"72a43e9d29f8bdc9d450c4031bd5b40f9f3c9b87137187da3b6f968e8fc891ae"} Oct 29 22:27:53.795623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.795266 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:27:53.835272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.835223 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" podStartSLOduration=1.835210368 podStartE2EDuration="1.835210368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:53.834253063 +0000 UTC m=+736.315570557" watchObservedRunningTime="2026-10-29 22:27:53.835210368 +0000 UTC m=+736.316527861" Oct 29 22:27:53.980194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.980155 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt"] Oct 29 22:27:53.984355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:53.984331 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.001333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.001301 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-8d3632-serving-cert\"" Oct 29 22:27:54.003063 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.003043 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt"] Oct 29 22:27:54.105720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.105636 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.105720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.105671 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-metrics-certs\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.105942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.105860 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.105998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.105955 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chz2\" (UniqueName: \"kubernetes.io/projected/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-kube-api-access-8chz2\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.155714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.155684 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:54.207315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.207284 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.207474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.207352 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chz2\" (UniqueName: \"kubernetes.io/projected/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-kube-api-access-8chz2\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.207474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.207424 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.207474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.207459 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-metrics-certs\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.208058 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.208026 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.209964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.209933 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-metrics-certs\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.210076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.210009 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.229628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.229607 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chz2\" (UniqueName: \"kubernetes.io/projected/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-kube-api-access-8chz2\") pod \"maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.298202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.298170 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:54.302810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.302779 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:54.441786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.441753 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:54.469999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.469970 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq"] Oct 29 22:27:54.475262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.475242 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:54.493574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.493552 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq"] Oct 29 22:27:54.547174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.547136 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpm8f\" (UniqueName: \"kubernetes.io/projected/b8ad7ba1-789d-4681-a532-39068330f54f-kube-api-access-fpm8f\") pod \"payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq\" (UID: \"b8ad7ba1-789d-4681-a532-39068330f54f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:54.547368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.547228 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ad7ba1-789d-4681-a532-39068330f54f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq\" (UID: \"b8ad7ba1-789d-4681-a532-39068330f54f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:54.548656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.548304 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-8d3632\"" Oct 29 22:27:54.593756 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.579733 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-8d3632-dockercfg-7mp7c\"" Oct 29 22:27:54.606761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.606655 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt"] Oct 29 22:27:54.609659 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:54.609632 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd1953c_f0a8_4353_bfe3_55e3d25c4212.slice/crio-2783abd20d7e5ba43985d3348eb1f8da0c3b904acd0f734b327f6877b188e02a WatchSource:0}: Error finding container 2783abd20d7e5ba43985d3348eb1f8da0c3b904acd0f734b327f6877b188e02a: Status 404 returned error can't find the container with id 2783abd20d7e5ba43985d3348eb1f8da0c3b904acd0f734b327f6877b188e02a Oct 29 22:27:54.649606 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.649574 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ad7ba1-789d-4681-a532-39068330f54f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq\" (UID: \"b8ad7ba1-789d-4681-a532-39068330f54f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:54.649738 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.649687 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpm8f\" (UniqueName: \"kubernetes.io/projected/b8ad7ba1-789d-4681-a532-39068330f54f-kube-api-access-fpm8f\") pod \"payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq\" (UID: \"b8ad7ba1-789d-4681-a532-39068330f54f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:54.650235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.650216 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ad7ba1-789d-4681-a532-39068330f54f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq\" (UID: \"b8ad7ba1-789d-4681-a532-39068330f54f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:54.667058 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.667032 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpm8f\" (UniqueName: \"kubernetes.io/projected/b8ad7ba1-789d-4681-a532-39068330f54f-kube-api-access-fpm8f\") pod \"payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq\" (UID: \"b8ad7ba1-789d-4681-a532-39068330f54f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:54.789074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.789042 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:54.805443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:54.805415 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" event={"ID":"4dd1953c-f0a8-4353-bfe3-55e3d25c4212","Type":"ContainerStarted","Data":"2783abd20d7e5ba43985d3348eb1f8da0c3b904acd0f734b327f6877b188e02a"} Oct 29 22:27:55.090248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.090220 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq"] Oct 29 22:27:55.101220 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:55.100859 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ad7ba1_789d_4681_a532_39068330f54f.slice/crio-739a85647d1741bace4206853fc716c5384c897318739ad9f2afa360ef014223 WatchSource:0}: Error finding container 739a85647d1741bace4206853fc716c5384c897318739ad9f2afa360ef014223: Status 404 returned error can't find the container with id 739a85647d1741bace4206853fc716c5384c897318739ad9f2afa360ef014223 Oct 29 22:27:55.127925 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.127849 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:55.199894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.199855 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6"] Oct 29 22:27:55.203851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.203820 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:55.216271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.216227 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-40cf9a\"" Oct 29 22:27:55.216411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.216366 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-40cf9a-dockercfg-vjgvl\"" Oct 29 22:27:55.228578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.228350 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6"] Oct 29 22:27:55.256172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.256145 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6\" (UID: \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:55.256294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.256210 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s8cr\" (UniqueName: \"kubernetes.io/projected/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-kube-api-access-6s8cr\") pod \"payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6\" (UID: \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:55.272740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.272704 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:55.357148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.357070 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6\" (UID: \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:55.357148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.357129 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s8cr\" (UniqueName: \"kubernetes.io/projected/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-kube-api-access-6s8cr\") pod \"payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6\" (UID: \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:55.357784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.357756 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6\" (UID: \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:55.374749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.374711 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s8cr\" (UniqueName: \"kubernetes.io/projected/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-kube-api-access-6s8cr\") pod \"payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6\" (UID: \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:55.538527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.538493 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:55.812529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.812492 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" event={"ID":"b8ad7ba1-789d-4681-a532-39068330f54f","Type":"ContainerStarted","Data":"1ef210596f270b6d091d2af346aae69be2b645708b3ebf5a4bcfeb010a6fabff"} Oct 29 22:27:55.813050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.812551 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" event={"ID":"b8ad7ba1-789d-4681-a532-39068330f54f","Type":"ContainerStarted","Data":"739a85647d1741bace4206853fc716c5384c897318739ad9f2afa360ef014223"} Oct 29 22:27:55.813050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.812578 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:27:55.814117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.814089 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" event={"ID":"4dd1953c-f0a8-4353-bfe3-55e3d25c4212","Type":"ContainerStarted","Data":"1f118e4053937b1bc3092ae63324df924c3366355eae5ba5f21447d1a152fec4"} Oct 29 22:27:55.814219 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.814138 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:27:55.846462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.846416 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" podStartSLOduration=1.8463748500000001 podStartE2EDuration="1.84637485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:55.845145935 +0000 UTC m=+738.326463430" watchObservedRunningTime="2026-10-29 22:27:55.84637485 +0000 UTC m=+738.327692347" Oct 29 22:27:55.882847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.882823 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6"] Oct 29 22:27:55.885058 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:55.885031 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f75d85_98e3_4f25_ba52_70895e9ac0fc.slice/crio-eb21a48bbb098afe153e3cb04d755df49c145f24519cd401c8a67d799812afd3 WatchSource:0}: Error finding container eb21a48bbb098afe153e3cb04d755df49c145f24519cd401c8a67d799812afd3: Status 404 returned error can't find the container with id eb21a48bbb098afe153e3cb04d755df49c145f24519cd401c8a67d799812afd3 Oct 29 22:27:55.885676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.885622 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" podStartSLOduration=2.4253877 podStartE2EDuration="2.885605075s" totalImagesPullingTime="460.217375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:55.884105567 +0000 UTC m=+738.365423061" watchObservedRunningTime="2026-10-29 22:27:55.885605075 +0000 UTC m=+738.366922568" Oct 29 22:27:55.892032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.892009 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz"] Oct 29 22:27:55.897521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.897498 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:55.909621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.909591 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-95672c-serving-cert\"" Oct 29 22:27:55.918603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.917926 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz"] Oct 29 22:27:55.963489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.963468 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvkfs\" (UniqueName: \"kubernetes.io/projected/eacc6253-bd0f-4197-b8c0-eab3a4732971-kube-api-access-mvkfs\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:55.963598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.963508 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc6253-bd0f-4197-b8c0-eab3a4732971-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:55.963598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.963526 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-metrics-certs\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:55.963729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:55.963704 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.021230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.021183 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:56.064841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.064811 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvkfs\" (UniqueName: \"kubernetes.io/projected/eacc6253-bd0f-4197-b8c0-eab3a4732971-kube-api-access-mvkfs\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.064985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.064852 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc6253-bd0f-4197-b8c0-eab3a4732971-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.064985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.064868 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-metrics-certs\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.064985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.064950 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.065116 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:27:56.065090 5610 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-serving-cert: secret "maas-api-e2e-worker-w3-95672c-serving-cert" not found Oct 29 22:27:56.065172 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:27:56.065160 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls podName:eacc6253-bd0f-4197-b8c0-eab3a4732971 nodeName:}" failed. No retries permitted until 2026-10-29 22:27:56.565139506 +0000 UTC m=+739.046457000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls") pod "maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" (UID: "eacc6253-bd0f-4197-b8c0-eab3a4732971") : secret "maas-api-e2e-worker-w3-95672c-serving-cert" not found Oct 29 22:27:56.065641 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.065620 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc6253-bd0f-4197-b8c0-eab3a4732971-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.067825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.067797 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-metrics-certs\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.083115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.083054 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvkfs\" (UniqueName: \"kubernetes.io/projected/eacc6253-bd0f-4197-b8c0-eab3a4732971-kube-api-access-mvkfs\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.163667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.163560 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:56.308434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.308367 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:56.583758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.583723 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.586510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.586483 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls\") pod \"maas-api-e2e-worker-w3-95672c-685684857c-xt2bz\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.625340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.625310 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:56.791008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.790964 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:56.817340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.817298 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:56.859545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.831418 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" event={"ID":"d6f75d85-98e3-4f25-ba52-70895e9ac0fc","Type":"ContainerStarted","Data":"959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4"} Oct 29 22:27:56.859545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.831473 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" event={"ID":"d6f75d85-98e3-4f25-ba52-70895e9ac0fc","Type":"ContainerStarted","Data":"eb21a48bbb098afe153e3cb04d755df49c145f24519cd401c8a67d799812afd3"} Oct 29 22:27:56.859545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:56.832031 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:27:57.169744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.169650 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" podStartSLOduration=2.169636091 podStartE2EDuration="2.169636091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:56.896363214 +0000 UTC m=+739.377680765" watchObservedRunningTime="2026-10-29 22:27:57.169636091 +0000 UTC m=+739.650953583" Oct 29 22:27:57.170174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.170160 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz"] Oct 29 22:27:57.173927 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:57.173901 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacc6253_bd0f_4197_b8c0_eab3a4732971.slice/crio-ff9e09649cd061fbcbb32b505e129e90bd8a773167d67349b8e63d0f5c527155 WatchSource:0}: Error finding container ff9e09649cd061fbcbb32b505e129e90bd8a773167d67349b8e63d0f5c527155: Status 404 returned error can't find the container with id ff9e09649cd061fbcbb32b505e129e90bd8a773167d67349b8e63d0f5c527155 Oct 29 22:27:57.669062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.669017 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp"] Oct 29 22:27:57.678566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.678542 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.683668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.683636 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp"] Oct 29 22:27:57.691626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.690721 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-879596-serving-cert\"" Oct 29 22:27:57.706438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.704456 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.706438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.704517 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb24q\" (UniqueName: \"kubernetes.io/projected/3940cff7-5a19-4fc8-9c07-9f8233186f2e-kube-api-access-qb24q\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.706438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.704576 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3940cff7-5a19-4fc8-9c07-9f8233186f2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.706438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.704664 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-metrics-certs\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.782143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.782109 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:57.805489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.805448 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-metrics-certs\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.805661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.805572 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.805661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.805622 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb24q\" (UniqueName: \"kubernetes.io/projected/3940cff7-5a19-4fc8-9c07-9f8233186f2e-kube-api-access-qb24q\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.805776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.805707 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3940cff7-5a19-4fc8-9c07-9f8233186f2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.806423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.806384 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3940cff7-5a19-4fc8-9c07-9f8233186f2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.808162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.808138 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-metrics-certs\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.808289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.808243 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.823035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.822960 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb24q\" (UniqueName: \"kubernetes.io/projected/3940cff7-5a19-4fc8-9c07-9f8233186f2e-kube-api-access-qb24q\") pod \"maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:57.836761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.836737 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" event={"ID":"eacc6253-bd0f-4197-b8c0-eab3a4732971","Type":"ContainerStarted","Data":"2c94420a6caa2e8f56548fa70995eaddd3fd7e9832c49759b544cd6bed52a7f0"} Oct 29 22:27:57.836873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.836792 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" event={"ID":"eacc6253-bd0f-4197-b8c0-eab3a4732971","Type":"ContainerStarted","Data":"ff9e09649cd061fbcbb32b505e129e90bd8a773167d67349b8e63d0f5c527155"} Oct 29 22:27:57.837001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.836983 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:27:57.871289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.871248 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" podStartSLOduration=2.480101076 podStartE2EDuration="2.871234687s" totalImagesPullingTime="391.133611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:57.870234909 +0000 UTC m=+740.351552405" watchObservedRunningTime="2026-10-29 22:27:57.871234687 +0000 UTC m=+740.352552217" Oct 29 22:27:57.913007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:57.912977 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:58.004986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:58.003633 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:58.175278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:58.175245 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:58.322907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:58.322866 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp"] Oct 29 22:27:58.323361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:58.323238 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:58.325435 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:27:58.325215 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3940cff7_5a19_4fc8_9c07_9f8233186f2e.slice/crio-c7335d8bc39398b4e7f72873e68e0c9509371b7c2a374ddddf2c188789ac1d05 WatchSource:0}: Error finding container c7335d8bc39398b4e7f72873e68e0c9509371b7c2a374ddddf2c188789ac1d05: Status 404 returned error can't find the container with id c7335d8bc39398b4e7f72873e68e0c9509371b7c2a374ddddf2c188789ac1d05 Oct 29 22:27:58.843463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:58.843433 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" event={"ID":"3940cff7-5a19-4fc8-9c07-9f8233186f2e","Type":"ContainerStarted","Data":"a938f95471ae6f9def363c910a62944dca927abdb4735f167c1820dda8801967"} Oct 29 22:27:58.843828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:58.843479 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" event={"ID":"3940cff7-5a19-4fc8-9c07-9f8233186f2e","Type":"ContainerStarted","Data":"c7335d8bc39398b4e7f72873e68e0c9509371b7c2a374ddddf2c188789ac1d05"} Oct 29 22:27:58.878315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:58.878273 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" podStartSLOduration=1.486460311 podStartE2EDuration="1.87825809s" totalImagesPullingTime="391.797779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:27:58.877350415 +0000 UTC m=+741.358667910" watchObservedRunningTime="2026-10-29 22:27:58.87825809 +0000 UTC m=+741.359575583" Oct 29 22:27:59.018792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:59.018591 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:59.174147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:59.174059 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:59.772587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:59.772553 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:27:59.848900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:59.848876 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:27:59.910409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:27:59.910375 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:00.076996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:00.076905 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:01.616069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.616028 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts"] Oct 29 22:28:01.621544 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.621522 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.637004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.636977 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-92fd86-serving-cert\"" Oct 29 22:28:01.648300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.648249 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts"] Oct 29 22:28:01.689412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.688429 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-maas-api-tls\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.689412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.688484 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-metrics-certs\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.689412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.688527 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qfl\" (UniqueName: \"kubernetes.io/projected/dc11cd91-f5b5-45d6-8504-2d56367411a9-kube-api-access-r2qfl\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.689412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.688573 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11cd91-f5b5-45d6-8504-2d56367411a9-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.790417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.789904 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-maas-api-tls\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.790417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.789959 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-metrics-certs\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.790417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.789999 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qfl\" (UniqueName: \"kubernetes.io/projected/dc11cd91-f5b5-45d6-8504-2d56367411a9-kube-api-access-r2qfl\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.790417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.790041 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11cd91-f5b5-45d6-8504-2d56367411a9-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.790981 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.790943 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11cd91-f5b5-45d6-8504-2d56367411a9-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.793443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.793418 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-maas-api-tls\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.793550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.793496 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-metrics-certs\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.818572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.818548 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qfl\" (UniqueName: \"kubernetes.io/projected/dc11cd91-f5b5-45d6-8504-2d56367411a9-kube-api-access-r2qfl\") pod \"maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:01.837719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.837695 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 22:28:01.853207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.853182 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:01.939585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:01.939485 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:02.001548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:02.000731 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:02.288103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:02.288069 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts"] Oct 29 22:28:02.292343 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:28:02.292316 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc11cd91_f5b5_45d6_8504_2d56367411a9.slice/crio-279f02a0bb8b3701b5df5b62039a79eac4beaef3dc765c7dafc00d2011a4334f WatchSource:0}: Error finding container 279f02a0bb8b3701b5df5b62039a79eac4beaef3dc765c7dafc00d2011a4334f: Status 404 returned error can't find the container with id 279f02a0bb8b3701b5df5b62039a79eac4beaef3dc765c7dafc00d2011a4334f Oct 29 22:28:02.947692 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:02.947647 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" event={"ID":"dc11cd91-f5b5-45d6-8504-2d56367411a9","Type":"ContainerStarted","Data":"7fe6e5fc0a1bb2b6d2e5750bf7ebdd52835e5591917c6b74a0dcf2d3205871fc"} Oct 29 22:28:02.948142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:02.947713 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" event={"ID":"dc11cd91-f5b5-45d6-8504-2d56367411a9","Type":"ContainerStarted","Data":"279f02a0bb8b3701b5df5b62039a79eac4beaef3dc765c7dafc00d2011a4334f"} Oct 29 22:28:02.948142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:02.947859 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:03.731324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:03.731296 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:03.848962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:03.848930 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 22:28:03.874415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:03.871272 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:03.911433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:03.896219 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" podStartSLOduration=2.568544155 podStartE2EDuration="2.896198343s" totalImagesPullingTime="327.654188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:02.993304243 +0000 UTC m=+745.474621737" watchObservedRunningTime="2026-10-29 22:28:03.896198343 +0000 UTC m=+746.377515836" Oct 29 22:28:04.809450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:04.809386 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 22:28:05.914421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:05.912991 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 22:28:06.229417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:06.221183 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:06.835033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:06.835002 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 22:28:08.552744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:08.552699 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:08.847575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:08.847491 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 22:28:08.993414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:08.989557 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 22:28:10.202744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.202700 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254"] Oct 29 22:28:10.219865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.219822 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.233260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.233183 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254"] Oct 29 22:28:10.239054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.239015 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-0d12c5-serving-cert\"" Oct 29 22:28:10.350710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.350673 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.350958 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.350936 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-metrics-certs\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.351178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.351162 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.351483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.351464 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdkff\" (UniqueName: \"kubernetes.io/projected/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-kube-api-access-bdkff\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.352195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.352098 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:10.455074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.454934 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.455257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.455089 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdkff\" (UniqueName: \"kubernetes.io/projected/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-kube-api-access-bdkff\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.455257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.455151 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.455257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.455176 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-metrics-certs\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.456990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.456965 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.513456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.491224 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdkff\" (UniqueName: \"kubernetes.io/projected/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-kube-api-access-bdkff\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.513724 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.492403 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:10.537418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.521933 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-metrics-certs\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.537418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.522413 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-maas-api-tls\") pod \"maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.553495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.553459 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:10.650429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:10.650376 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:11.010428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:11.010187 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:11.010645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:11.010616 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254"] Oct 29 22:28:11.010795 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:28:11.010622 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896bef94_faf6_4ff5_b7b6_4c3dc5d09b3a.slice/crio-2dc138fb4bd726fcdb985db0dce171ee7a61271e15ab33d96d584cb1bf1f40f1 WatchSource:0}: Error finding container 2dc138fb4bd726fcdb985db0dce171ee7a61271e15ab33d96d584cb1bf1f40f1: Status 404 returned error can't find the container with id 2dc138fb4bd726fcdb985db0dce171ee7a61271e15ab33d96d584cb1bf1f40f1 Oct 29 22:28:11.164377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:11.164331 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" event={"ID":"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a","Type":"ContainerStarted","Data":"2dc138fb4bd726fcdb985db0dce171ee7a61271e15ab33d96d584cb1bf1f40f1"} Oct 29 22:28:11.615826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:11.615795 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:11.759347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:11.759300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:11.919437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:11.913118 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:12.174822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:12.174731 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" event={"ID":"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a","Type":"ContainerStarted","Data":"f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8"} Oct 29 22:28:12.175676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:12.175652 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:12.212343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:12.212274 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" podStartSLOduration=1.835667256 podStartE2EDuration="2.21225428s" totalImagesPullingTime="376.587024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:12.21080712 +0000 UTC m=+754.692124614" watchObservedRunningTime="2026-10-29 22:28:12.21225428 +0000 UTC m=+754.693571774" Oct 29 22:28:12.350175 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:12.350132 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:13.736528 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:28:13.736491 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:28:13.893679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:13.893638 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:14.193449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:14.193375 5610 generic.go:356] "Generic (PLEG): container finished" podID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerID="43a669058423f61f015957066ca288335db15e39653dec0a834d736fba87aa69" exitCode=0 Oct 29 22:28:14.193668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:14.193519 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" event={"ID":"295069a5-7550-4e03-b1aa-f6c7ae3aa310","Type":"ContainerDied","Data":"43a669058423f61f015957066ca288335db15e39653dec0a834d736fba87aa69"} Oct 29 22:28:14.194134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:14.194111 5610 scope.go:122] "RemoveContainer" containerID="43a669058423f61f015957066ca288335db15e39653dec0a834d736fba87aa69" Oct 29 22:28:14.195895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:14.195858 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:14.196005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:14.195952 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:14.196074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:14.196002 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:15.045614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.045582 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:15.214433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.214371 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" event={"ID":"295069a5-7550-4e03-b1aa-f6c7ae3aa310","Type":"ContainerStarted","Data":"39656155f4fd52a1c2dc37b239593b31bacc3517449a71619541d94f55924d50"} Oct 29 22:28:15.218192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.218158 5610 generic.go:356] "Generic (PLEG): container finished" podID="573f8be8-460f-4353-b9ee-e85751503f91" containerID="0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1" exitCode=0 Oct 29 22:28:15.218364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.218242 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" event={"ID":"573f8be8-460f-4353-b9ee-e85751503f91","Type":"ContainerDied","Data":"0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1"} Oct 29 22:28:15.218655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.218636 5610 scope.go:122] "RemoveContainer" containerID="0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1" Oct 29 22:28:15.220334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.220303 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:15.220459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.220424 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:15.220527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.220472 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:15.224513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.222821 5610 generic.go:356] "Generic (PLEG): container finished" podID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerID="e34ea2dd206d860bd88724660128aadffa7eb4e64d4da40194e6e13b33de9f63" exitCode=0 Oct 29 22:28:15.224513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.222918 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" event={"ID":"eca4a254-88f7-40a2-876c-6dc36cfd9259","Type":"ContainerDied","Data":"e34ea2dd206d860bd88724660128aadffa7eb4e64d4da40194e6e13b33de9f63"} Oct 29 22:28:15.224513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.223260 5610 scope.go:122] "RemoveContainer" containerID="e34ea2dd206d860bd88724660128aadffa7eb4e64d4da40194e6e13b33de9f63" Oct 29 22:28:15.226445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.226418 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:15.226686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.226612 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:15.226859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.226829 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:15.617127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.617041 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:28:15.617517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.617374 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:28:15.617698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:15.617675 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 22:28:16.105081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:16.105048 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:16.230286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:16.230256 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" event={"ID":"573f8be8-460f-4353-b9ee-e85751503f91","Type":"ContainerStarted","Data":"36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401"} Oct 29 22:28:16.232308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:16.232283 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" event={"ID":"eca4a254-88f7-40a2-876c-6dc36cfd9259","Type":"ContainerStarted","Data":"fc58c540772a8cbec404d24dcce425300a036cdd7e650fb20d73361ce46b3122"} Oct 29 22:28:16.347309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:16.344274 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:28:16.347309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:16.345459 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:28:16.347309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:16.345544 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 22:28:16.618418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:16.617782 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:28:16.618418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:16.617858 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 22:28:17.054378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.054278 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:28:17.055266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.055241 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:28:17.055330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.055290 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 22:28:17.350474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.348924 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:28:17.350474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.349000 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 22:28:17.582417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.581854 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:17.618466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.617786 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:28:17.618466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.617846 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 22:28:17.750420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:17.745104 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:18.054417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.054335 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:28:18.054417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.054410 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 22:28:18.200415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.200260 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:18.331470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.322999 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2"] Oct 29 22:28:18.349741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.348659 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.360151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.359735 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:28:18.360151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.359799 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 22:28:18.397430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.394380 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-0d12c5-openshift-default-dockercfg-477sq\"" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402591 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402660 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402699 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402729 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6xc\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-kube-api-access-sz6xc\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402792 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f583580-6274-4723-8d24-931c393a76c4-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402837 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402873 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f583580-6274-4723-8d24-931c393a76c4-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402943 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.449416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.402981 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.483256 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2"] Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.485886 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515324 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515378 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515436 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6xc\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-kube-api-access-sz6xc\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515502 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f583580-6274-4723-8d24-931c393a76c4-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515556 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515591 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f583580-6274-4723-8d24-931c393a76c4-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515664 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515694 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.515760 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.517479 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f583580-6274-4723-8d24-931c393a76c4-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.517729 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.517958 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.518199 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.579455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.537683 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.657472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.655216 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f583580-6274-4723-8d24-931c393a76c4-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.657472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.655488 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.657472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.657202 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:28:18.657472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.657257 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 22:28:18.708425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.695941 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6xc\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-kube-api-access-sz6xc\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.735875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.735837 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:18.736524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:18.736443 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:19.026892 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.026606 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:19.055162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.055125 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:28:19.055291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.055187 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 22:28:19.342473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.338718 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 22:28:19.350447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.343706 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:28:19.350447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.343768 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 22:28:19.458846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.458707 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2"] Oct 29 22:28:19.555326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.555286 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:19.555464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.555367 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:19.555464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.555426 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:28:19.653555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.653449 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:28:19.653828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.653796 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 22:28:19.832417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:19.828994 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:28:20.060838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.060802 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:28:20.061086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.061059 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 22:28:20.343802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.343739 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:28:20.344003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.343811 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 22:28:20.406527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.406483 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" event={"ID":"1f583580-6274-4723-8d24-931c393a76c4","Type":"ContainerStarted","Data":"3c94d9c03e2b7ca03f6461a9c9218d477707f829c7ff264a8ea98db918401ebe"} Oct 29 22:28:20.406828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.406805 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" event={"ID":"1f583580-6274-4723-8d24-931c393a76c4","Type":"ContainerStarted","Data":"b8b5d8886231dac756cab6da92efe5f7daa0391155bd0cd4a26eb77b28287537"} Oct 29 22:28:20.456740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.456692 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podStartSLOduration=2.456668599 podStartE2EDuration="2.456668599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:28:20.455340293 +0000 UTC m=+762.936657787" watchObservedRunningTime="2026-10-29 22:28:20.456668599 +0000 UTC m=+762.937986092" Oct 29 22:28:20.617185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.617097 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 29 22:28:20.617807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.617773 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 29 22:28:20.783418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.780503 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:28:20.792418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:20.788483 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:28:21.029978 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.029926 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:21.030297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.030045 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:28:21.030297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.030088 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:28:21.054985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.054939 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:28:21.055299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.055260 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 22:28:21.343322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.343238 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 29 22:28:21.343322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.343290 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 29 22:28:21.441216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.441182 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:21.581438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.579468 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:21.654473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.652094 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:28:21.654473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:21.653470 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:28:22.060030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.056447 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:28:22.060030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.056521 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:28:22.083415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.076383 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 29 22:28:22.083415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.076466 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 29 22:28:22.197302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.197267 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:22.377729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.377642 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:22.553725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.553672 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:28:22.555195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.555146 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 22:28:22.919871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:22.919838 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:23.027572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:23.027541 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:28:23.027751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:23.027593 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:28:23.076430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:23.064644 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:28:23.078091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:23.078064 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:28:23.079247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:23.079033 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:23.682418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:23.680505 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:24.028190 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:24.028148 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:28:24.028695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:24.028217 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:28:25.028059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:25.028021 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:28:25.028271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:25.028088 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:28:25.374069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:25.370642 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:26.028292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:26.028258 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:28:26.028719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:26.028309 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:28:26.316165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:26.316059 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:26.754739 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:26.754696 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:26.933376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:26.933340 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:27.028004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:27.027846 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 29 22:28:27.028004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:27.027913 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 29 22:28:27.085166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:27.085114 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:27.221013 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:27.220982 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:28.031801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:28.031773 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:28.032071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:28.032052 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:28.032685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:28.032666 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:28:28.294912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:28.294829 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:28.548069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:28.547883 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:30.119947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:30.119875 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:31.271674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:31.271636 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:31.578051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:31.577958 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:32.550074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:32.549995 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:33.164811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:33.164770 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:33.341411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:33.336326 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:33.495679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:33.495639 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:34.823122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:34.823089 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:35.876349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:35.876309 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:36.325206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:36.325154 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:37.421646 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:37.421595 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:38.194846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:38.194810 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:39.704938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:39.704909 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:40.819582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:40.819545 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:41.172969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:41.172883 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:42.743386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:42.743354 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:43.709459 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:28:43.709422 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:28:44.172259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:44.172171 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:45.901489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:45.901443 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:47.249826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:47.249795 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:47.442525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:47.442492 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:49.138722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:49.138683 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:50.660069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:50.660027 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:52.082419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:52.076344 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:53.559190 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:53.559161 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:53.695269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:53.695214 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:55.063878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:55.063842 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:56.646145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:56.646097 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:58.218596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:58.218552 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:28:59.727856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:28:59.727825 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:01.406505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:01.406409 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:02.718005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:02.717960 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:02.896153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:02.896122 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:04.032444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:04.032413 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:04.750422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:04.748078 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:06.230642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:06.230611 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:07.720220 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:07.720178 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:08.213912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:08.213875 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:09.428099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:09.428067 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:10.392285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:10.392248 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:10.997424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:10.997330 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:12.447042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:12.446954 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:13.710869 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:29:13.710832 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:29:14.036285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:14.036215 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:15.555324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:15.555288 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:16.609939 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:16.609876 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:16.829544 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:16.829507 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:17.327419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.325145 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:17.850236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.850204 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk"] Oct 29 22:29:17.854069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.854048 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.872364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.872339 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4a0287-openshift-default-dockercfg-dcw9p\"" Oct 29 22:29:17.884383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.884359 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk"] Oct 29 22:29:17.951145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951112 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.951314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951154 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.951383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951310 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.951383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951358 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.951511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951477 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.951581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951515 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dh9j\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-kube-api-access-7dh9j\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.951640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951575 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.951696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951665 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:17.951745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:17.951705 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.052930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.052894 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.052930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.052935 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.052988 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053016 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053155 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053191 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dh9j\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-kube-api-access-7dh9j\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053347 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053388 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053482 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053485 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053498 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053539 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053731 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.053810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.053788 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.055710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.055683 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.055866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.055848 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.072877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.072849 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.073030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.072956 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dh9j\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-kube-api-access-7dh9j\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.210164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.210130 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:18.524292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.523901 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk"] Oct 29 22:29:18.526677 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:29:18.526645 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50172ea_80ec_49ee_8534_f8f9841a4fc9.slice/crio-c122f7139515deea38f89ebed7439aa05a6bba0b7e48966f42e7aecc23c8e4ae WatchSource:0}: Error finding container c122f7139515deea38f89ebed7439aa05a6bba0b7e48966f42e7aecc23c8e4ae: Status 404 returned error can't find the container with id c122f7139515deea38f89ebed7439aa05a6bba0b7e48966f42e7aecc23c8e4ae Oct 29 22:29:18.529341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.529307 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:18.529486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.529388 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:18.529554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.529502 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:18.925514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.925383 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" event={"ID":"d50172ea-80ec-49ee-8534-f8f9841a4fc9","Type":"ContainerStarted","Data":"46bd259206ee3467dc4e64e7c803b98db1c684fb69456dbf88c739204d0f9d20"} Oct 29 22:29:18.925514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.925474 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" event={"ID":"d50172ea-80ec-49ee-8534-f8f9841a4fc9","Type":"ContainerStarted","Data":"c122f7139515deea38f89ebed7439aa05a6bba0b7e48966f42e7aecc23c8e4ae"} Oct 29 22:29:18.994674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.994643 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:18.994959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:18.994914 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podStartSLOduration=1.994899964 podStartE2EDuration="1.994899964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:18.994217199 +0000 UTC m=+821.475534694" watchObservedRunningTime="2026-10-29 22:29:18.994899964 +0000 UTC m=+821.476217456" Oct 29 22:29:19.159886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:19.159856 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:19.211021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:19.210976 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:19.212884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:19.212855 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 29 22:29:19.213004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:19.212916 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 29 22:29:20.211011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:20.210918 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 29 22:29:20.211011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:20.210991 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 29 22:29:20.696313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:20.696268 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:21.210980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:21.210949 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 29 22:29:21.211139 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:21.211021 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 29 22:29:22.211522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:22.211486 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 29 22:29:22.211912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:22.211561 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 29 22:29:22.593606 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:22.593505 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:23.210532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:23.210483 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 29 22:29:23.210733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:23.210550 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 29 22:29:24.211338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:24.211290 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 29 22:29:24.211869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:24.211372 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 29 22:29:24.238118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:24.238090 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:25.214932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:25.214902 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:25.215422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:25.215299 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:25.215898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:25.215879 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:29:25.670448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:25.667435 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:25.963466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:25.963385 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:27.730547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:27.729555 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:29.439981 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:29.439942 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:30.244169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.244130 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz"] Oct 29 22:29:30.247340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.247317 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.265026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.264994 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-home\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.265151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.265051 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhtcg\" (UniqueName: \"kubernetes.io/projected/6f0a67a0-a252-4a00-94c5-cc659b9a877b-kube-api-access-qhtcg\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.265151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.265138 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-model-cache\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.265280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.265241 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tls-certs\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.265280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.265267 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tmp-dir\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.265386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.265361 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-dshm\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.281555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.281530 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"kube-root-ca.crt\"" Oct 29 22:29:30.281698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.281561 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"openshift-service-ca.crt\"" Oct 29 22:29:30.281698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.281648 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"premium-w5-0d12c5-kserve-self-signed-certs\"" Oct 29 22:29:30.281971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.281952 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-0d12c5\"/\"default-dockercfg-9x74l\"" Oct 29 22:29:30.283636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.283613 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz"] Oct 29 22:29:30.366062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366035 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tls-certs\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.366221 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366104 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tmp-dir\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.366221 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366175 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-dshm\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.366344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366247 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:30.366344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366317 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-home\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.366489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366375 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhtcg\" (UniqueName: \"kubernetes.io/projected/6f0a67a0-a252-4a00-94c5-cc659b9a877b-kube-api-access-qhtcg\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.366489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366443 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-model-cache\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.366801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366764 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-home\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.366801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366786 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tmp-dir\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.367005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.366822 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-model-cache\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.368433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.368408 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-dshm\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.368673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.368658 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tls-certs\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.385896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.385875 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhtcg\" (UniqueName: \"kubernetes.io/projected/6f0a67a0-a252-4a00-94c5-cc659b9a877b-kube-api-access-qhtcg\") pod \"premium-w5-0d12c5-kserve-5dc846f5fb-czndz\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.570118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.570039 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:30.873677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:30.873647 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz"] Oct 29 22:29:30.875651 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:29:30.875627 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f0a67a0_a252_4a00_94c5_cc659b9a877b.slice/crio-c2269e9c37bef97438fad47396d73d166c06ef3f11d5115f5cb02481a99ee73c WatchSource:0}: Error finding container c2269e9c37bef97438fad47396d73d166c06ef3f11d5115f5cb02481a99ee73c: Status 404 returned error can't find the container with id c2269e9c37bef97438fad47396d73d166c06ef3f11d5115f5cb02481a99ee73c Oct 29 22:29:31.009699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:31.009643 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:31.021325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:31.021285 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" event={"ID":"6f0a67a0-a252-4a00-94c5-cc659b9a877b","Type":"ContainerStarted","Data":"1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac"} Oct 29 22:29:31.021633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:31.021601 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" event={"ID":"6f0a67a0-a252-4a00-94c5-cc659b9a877b","Type":"ContainerStarted","Data":"c2269e9c37bef97438fad47396d73d166c06ef3f11d5115f5cb02481a99ee73c"} Oct 29 22:29:31.068636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:31.068570 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podStartSLOduration=1.06855592 podStartE2EDuration="1.06855592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:31.067096461 +0000 UTC m=+833.548413955" watchObservedRunningTime="2026-10-29 22:29:31.06855592 +0000 UTC m=+833.549873470" Oct 29 22:29:31.570897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:31.570861 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:32.739496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.739446 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:32.792423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.792379 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk"] Oct 29 22:29:32.795780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.795755 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.812546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.812527 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8d3632-openshift-default-dockercfg-x5w92\"" Oct 29 22:29:32.826750 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.826728 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk"] Oct 29 22:29:32.906337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906294 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.906542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906347 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.906542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906525 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.906653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906563 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.906653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906633 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.906743 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906691 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.906825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906809 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.906882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906837 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:32.906882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:32.906869 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-kube-api-access-zq8zl\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.007766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.007686 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.007766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.007727 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.007766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.007748 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.007795 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.007863 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.007884 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.007914 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-kube-api-access-zq8zl\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.007982 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.008016 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.008150 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.008293 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.008420 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.008778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.008723 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.009607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.009584 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.010844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.010819 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.011004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.010976 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.042425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.042373 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.043293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.043275 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-kube-api-access-zq8zl\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.112275 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.112243 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:33.444307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.444281 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk"] Oct 29 22:29:33.445736 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:29:33.445704 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae8da64_61ec_463f_b44f_299bd4c6d1b4.slice/crio-2696042730eac3f54a2ff1f48e720ba044001f56f791da21f8263a7020cd7354 WatchSource:0}: Error finding container 2696042730eac3f54a2ff1f48e720ba044001f56f791da21f8263a7020cd7354: Status 404 returned error can't find the container with id 2696042730eac3f54a2ff1f48e720ba044001f56f791da21f8263a7020cd7354 Oct 29 22:29:33.448165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.448137 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:33.448234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.448201 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:33.448275 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:33.448231 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:34.053271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:34.050900 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" event={"ID":"9ae8da64-61ec-463f-b44f-299bd4c6d1b4","Type":"ContainerStarted","Data":"d01c5755e5f00609f395119461e1ba37dd2d76c715c2d27a2ec7524b19db22ca"} Oct 29 22:29:34.053271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:34.050965 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" event={"ID":"9ae8da64-61ec-463f-b44f-299bd4c6d1b4","Type":"ContainerStarted","Data":"2696042730eac3f54a2ff1f48e720ba044001f56f791da21f8263a7020cd7354"} Oct 29 22:29:34.113382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:34.113347 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:34.115014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:34.114978 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 29 22:29:34.115150 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:34.115064 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 29 22:29:34.124250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:34.124201 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podStartSLOduration=2.124185528 podStartE2EDuration="2.124185528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:34.112202993 +0000 UTC m=+836.593520488" watchObservedRunningTime="2026-10-29 22:29:34.124185528 +0000 UTC m=+836.605503021" Oct 29 22:29:34.434467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:34.431254 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:35.112895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:35.112858 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 29 22:29:35.113287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:35.112911 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 29 22:29:36.039960 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:36.039921 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:36.112819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:36.112789 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 29 22:29:36.113042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:36.113021 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 29 22:29:36.387607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:36.380276 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:37.114816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:37.114779 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 29 22:29:37.115549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:37.115516 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 29 22:29:38.112822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:38.112783 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 29 22:29:38.112996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:38.112848 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 29 22:29:39.112839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:39.112798 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 29 22:29:39.113677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:39.112864 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 29 22:29:39.219420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:39.216199 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:40.112610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:40.112567 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 29 22:29:40.112819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:40.112634 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 29 22:29:40.570933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:40.570902 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:40.583867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:40.583842 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:41.042425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:41.042375 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:41.132760 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:41.132730 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:41.134651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:41.134631 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:41.142760 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:41.142731 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:29:41.164420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:41.163619 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:29:42.653090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:42.653053 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:43.712307 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:29:43.712263 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:29:44.431408 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:44.431328 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:46.142525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:46.142482 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:47.816455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:47.816380 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:48.212475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.212434 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn"] Oct 29 22:29:48.217091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.217061 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.232974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.232941 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-92fd86-openshift-default-dockercfg-485tp\"" Oct 29 22:29:48.247693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.247657 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn"] Oct 29 22:29:48.297424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297377 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.297608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297429 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7a8eca-7072-46d9-964f-145ef0530128-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.297608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297549 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdszl\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-kube-api-access-fdszl\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.297722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297642 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.297722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297662 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.297836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297741 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.297836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297760 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.297954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297840 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.297954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.297905 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7a8eca-7072-46d9-964f-145ef0530128-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.398464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398428 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.398464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398466 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.398740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398495 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.398740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398537 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7a8eca-7072-46d9-964f-145ef0530128-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.398740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398586 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.398740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398616 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7a8eca-7072-46d9-964f-145ef0530128-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.398740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398672 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdszl\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-kube-api-access-fdszl\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.398740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398725 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.399051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398749 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.399051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.398920 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.399221 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.399200 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.399369 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.399348 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.399528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.399445 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7a8eca-7072-46d9-964f-145ef0530128-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.399611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.399583 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.401088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.401064 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.401753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.401732 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7a8eca-7072-46d9-964f-145ef0530128-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.419059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.419030 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.419166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.419136 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdszl\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-kube-api-access-fdszl\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.531151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.531065 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:48.843362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.843331 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn"] Oct 29 22:29:48.845224 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:29:48.845198 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7a8eca_7072_46d9_964f_145ef0530128.slice/crio-ff4d6db072e6803ba1dcfb007dbfb8614d0333d090a6ee4b94208648c0153eec WatchSource:0}: Error finding container ff4d6db072e6803ba1dcfb007dbfb8614d0333d090a6ee4b94208648c0153eec: Status 404 returned error can't find the container with id ff4d6db072e6803ba1dcfb007dbfb8614d0333d090a6ee4b94208648c0153eec Oct 29 22:29:48.847871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.847844 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:48.847963 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.847905 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:48.847963 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:48.847936 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:29:49.197894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:49.197851 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" event={"ID":"6a7a8eca-7072-46d9-964f-145ef0530128","Type":"ContainerStarted","Data":"228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea"} Oct 29 22:29:49.198080 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:49.197934 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" event={"ID":"6a7a8eca-7072-46d9-964f-145ef0530128","Type":"ContainerStarted","Data":"ff4d6db072e6803ba1dcfb007dbfb8614d0333d090a6ee4b94208648c0153eec"} Oct 29 22:29:49.245614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:49.245554 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podStartSLOduration=1.245534772 podStartE2EDuration="1.245534772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:29:49.242251999 +0000 UTC m=+851.723569492" watchObservedRunningTime="2026-10-29 22:29:49.245534772 +0000 UTC m=+851.726852265" Oct 29 22:29:49.543427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:49.543313 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:49.548454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:49.544902 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:49.548454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:49.544957 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 22:29:49.587416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:49.580381 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:50.532507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:50.532467 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:50.532961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:50.532535 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 22:29:51.272205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:51.272138 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:51.531795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:51.531712 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:51.531795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:51.531769 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 22:29:52.531783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:52.531751 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:52.532197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:52.531818 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 22:29:53.022208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:53.022078 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:53.532505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:53.532472 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:53.532900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:53.532535 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 22:29:54.532280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:54.532244 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 29 22:29:54.532497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:54.532299 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 29 22:29:54.589085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:54.582370 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:55.537173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:55.537144 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:55.537566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:55.537545 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:55.538296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:55.538277 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:29:56.309424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:56.308945 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:57.080157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:57.080123 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:29:57.884460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:29:57.884418 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:00.495024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:00.490935 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:02.662233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:02.662194 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:03.180379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.180325 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll"] Oct 29 22:30:03.185337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.185315 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.215162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.215131 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll"] Oct 29 22:30:03.263063 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263027 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qx2\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-kube-api-access-d4qx2\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.263206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263103 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.263265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263204 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.263299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263258 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.263332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263314 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.263418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263379 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f46e04-00bb-4620-872b-cc71207bc368-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.263474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263444 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f46e04-00bb-4620-872b-cc71207bc368-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.263522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263482 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.263522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.263516 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.364386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364353 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qx2\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-kube-api-access-d4qx2\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.364586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364459 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.364586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364484 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.364586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364510 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.364586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364533 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.364855 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364827 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f46e04-00bb-4620-872b-cc71207bc368-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.364962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364918 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f46e04-00bb-4620-872b-cc71207bc368-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.365043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364969 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.365043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.365007 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.365043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.364919 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.365514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.365459 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.365636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.365620 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f46e04-00bb-4620-872b-cc71207bc368-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.366142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.365927 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.366443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.366419 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.367475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.367428 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.368013 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.367992 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f46e04-00bb-4620-872b-cc71207bc368-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.388567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.388540 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qx2\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-kube-api-access-d4qx2\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.390067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.390042 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.504206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.504174 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:03.835637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.835608 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll"] Oct 29 22:30:03.837963 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:30:03.837937 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f46e04_00bb_4620_872b_cc71207bc368.slice/crio-bac3a6eb3497255467543861801dc08833a6fda1469e9101c66297bda2a9f737 WatchSource:0}: Error finding container bac3a6eb3497255467543861801dc08833a6fda1469e9101c66297bda2a9f737: Status 404 returned error can't find the container with id bac3a6eb3497255467543861801dc08833a6fda1469e9101c66297bda2a9f737 Oct 29 22:30:03.840427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.840371 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:30:03.840497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.840465 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:30:03.840546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:03.840497 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:30:04.651714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:04.651675 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" event={"ID":"92f46e04-00bb-4620-872b-cc71207bc368","Type":"ContainerStarted","Data":"bda7ba7967d5926c2ee8056e18d1c57a57f70f2b33a86637a2948d152f2bc83b"} Oct 29 22:30:04.651897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:04.651741 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" event={"ID":"92f46e04-00bb-4620-872b-cc71207bc368","Type":"ContainerStarted","Data":"bac3a6eb3497255467543861801dc08833a6fda1469e9101c66297bda2a9f737"} Oct 29 22:30:04.795471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:04.795257 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" podStartSLOduration=1.7951128189999999 podStartE2EDuration="1.795112819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:04.758188352 +0000 UTC m=+867.239505837" watchObservedRunningTime="2026-10-29 22:30:04.795112819 +0000 UTC m=+867.276430312" Oct 29 22:30:04.815417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:04.806941 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:05.504887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:05.504855 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:05.506524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:05.506496 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 22:30:05.506695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:05.506549 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 22:30:06.505133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:06.505093 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 22:30:06.505592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:06.505152 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 22:30:06.745539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:06.745502 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:06.954900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:06.954865 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr"] Oct 29 22:30:06.967728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:06.967700 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:06.991454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:06.991422 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr"] Oct 29 22:30:06.998932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:06.998904 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:30:06.999160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:06.999135 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 22:30:07.036433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.036388 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.036615 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.036588 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.036721 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.036702 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qct\" (UniqueName: \"kubernetes.io/projected/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-kube-api-access-b2qct\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.138250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.138211 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.138467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.138307 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.138467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.138346 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qct\" (UniqueName: \"kubernetes.io/projected/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-kube-api-access-b2qct\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.139095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.139070 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.140707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.140677 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.163430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.159815 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qct\" (UniqueName: \"kubernetes.io/projected/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-kube-api-access-b2qct\") pod \"maas-api-revoke-sub-28c5630c-g2mvr\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.299103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.299009 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:07.505034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.504999 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 22:30:07.505230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.505057 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 22:30:07.632915 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:30:07.632883 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb70b1f_546f_407c_a7f5_6dc0629c2e0b.slice/crio-909f2d3ed7909c8dacbeacc437bf605de7196abff762332d5e0d8c0d907be0df WatchSource:0}: Error finding container 909f2d3ed7909c8dacbeacc437bf605de7196abff762332d5e0d8c0d907be0df: Status 404 returned error can't find the container with id 909f2d3ed7909c8dacbeacc437bf605de7196abff762332d5e0d8c0d907be0df Oct 29 22:30:07.638231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.638207 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr"] Oct 29 22:30:07.694598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:07.694558 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" event={"ID":"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b","Type":"ContainerStarted","Data":"909f2d3ed7909c8dacbeacc437bf605de7196abff762332d5e0d8c0d907be0df"} Oct 29 22:30:08.509886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:08.509853 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 22:30:08.510527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:08.510495 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 22:30:08.807469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:08.806129 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:09.505038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:09.504998 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 29 22:30:09.505243 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:09.505055 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 29 22:30:10.382822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:10.382741 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:10.709020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:10.708983 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:10.709656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:10.709629 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:10.711013 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:10.710955 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:30:11.722285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:11.722199 5610 generic.go:356] "Generic (PLEG): container finished" podID="bfb70b1f-546f-407c-a7f5-6dc0629c2e0b" containerID="64692eaa642e17626b136a11c78b300db19070c93f7773d025945e06dc52fc32" exitCode=0 Oct 29 22:30:11.722669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:11.722289 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" event={"ID":"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b","Type":"ContainerDied","Data":"64692eaa642e17626b136a11c78b300db19070c93f7773d025945e06dc52fc32"} Oct 29 22:30:12.001521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:12.001376 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:12.876471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:12.876446 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:12.950517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:12.950488 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-kube-api-access-b2qct\" (UniqueName: \"kubernetes.io/projected/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-kube-api-access-b2qct\") pod \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " Oct 29 22:30:12.950711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:12.950621 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-xks-ca\") pod \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " Oct 29 22:30:12.950711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:12.950691 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-openshift-ca\") pod \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\" (UID: \"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b\") " Oct 29 22:30:12.951329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:12.951279 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-openshift-ca" pod "bfb70b1f-546f-407c-a7f5-6dc0629c2e0b" (UID: "bfb70b1f-546f-407c-a7f5-6dc0629c2e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:30:12.953054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:12.953020 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-xks-ca" pod "bfb70b1f-546f-407c-a7f5-6dc0629c2e0b" (UID: "bfb70b1f-546f-407c-a7f5-6dc0629c2e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:30:12.953155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:12.953118 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-kube-api-access-b2qct" pod "bfb70b1f-546f-407c-a7f5-6dc0629c2e0b" (UID: "bfb70b1f-546f-407c-a7f5-6dc0629c2e0b"). InnerVolumeSpecName "kube-api-access-b2qct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:30:13.052573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:13.052496 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-xks-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:30:13.052573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:13.052522 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-maas-api-openshift-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:30:13.052573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:13.052533 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qct\" (UniqueName: \"kubernetes.io/projected/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b-kube-api-access-b2qct\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:30:13.753418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:13.752583 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" event={"ID":"bfb70b1f-546f-407c-a7f5-6dc0629c2e0b","Type":"ContainerDied","Data":"909f2d3ed7909c8dacbeacc437bf605de7196abff762332d5e0d8c0d907be0df"} Oct 29 22:30:13.753418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:13.752646 5610 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909f2d3ed7909c8dacbeacc437bf605de7196abff762332d5e0d8c0d907be0df" Oct 29 22:30:13.753418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:13.752879 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr" Oct 29 22:30:13.757415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:13.753783 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:13.758767 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:30:13.758708 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:30:17.420418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:17.407581 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:19.486978 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:19.486939 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:21.341946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:21.341896 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:22.229420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:22.229376 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:22.916788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:22.916755 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:24.621864 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:24.621823 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:26.786242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:26.786169 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:28.342052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:28.342022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:29.935383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:29.935348 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:31.411053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:31.410930 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:32.862648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:32.862613 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:33.063091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.063057 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg"] Oct 29 22:30:33.063719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.063703 5610 cpu_manager.go:414] "containerMap: removing container" podUID="bfb70b1f-546f-407c-a7f5-6dc0629c2e0b" containerName="revoke-keys" Oct 29 22:30:33.063781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.063723 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb70b1f-546f-407c-a7f5-6dc0629c2e0b" containerName="revoke-keys" Oct 29 22:30:33.063860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.063848 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb70b1f-546f-407c-a7f5-6dc0629c2e0b" containerName="revoke-keys" Oct 29 22:30:33.068730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.068684 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.090587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.090560 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg"] Oct 29 22:30:33.220199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220163 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.220199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220202 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.220447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220230 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.220503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220453 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.220503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220490 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.220639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220616 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.220698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220676 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.220791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220767 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.220842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.220813 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlm6f\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-kube-api-access-vlm6f\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322084 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322115 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322160 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322200 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322274 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322317 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlm6f\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-kube-api-access-vlm6f\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322438 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322468 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322519 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322539 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.322911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322884 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.323101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.322922 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.323225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.323149 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.323225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.323199 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.325100 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.325073 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.325694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.325668 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.339249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.339219 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlm6f\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-kube-api-access-vlm6f\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.339518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.339496 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.384532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.384499 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:33.712254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.712231 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg"] Oct 29 22:30:33.713743 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:30:33.713720 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0341c0e9_195b_4953_bcf4_e5f34f7949d9.slice/crio-ede8cdec8887fe556d49f10308ac82661dac89710c55b5242aa558b7e4297601 WatchSource:0}: Error finding container ede8cdec8887fe556d49f10308ac82661dac89710c55b5242aa558b7e4297601: Status 404 returned error can't find the container with id ede8cdec8887fe556d49f10308ac82661dac89710c55b5242aa558b7e4297601 Oct 29 22:30:33.716123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.716094 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:30:33.716206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.716155 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:30:33.716206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:33.716184 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:30:34.067031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:34.066923 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" event={"ID":"0341c0e9-195b-4953-bcf4-e5f34f7949d9","Type":"ContainerStarted","Data":"52d035c53772f65f13487294cb042e7097a061bc150199f45157d10a2d2455e4"} Oct 29 22:30:34.067647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:34.067032 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" event={"ID":"0341c0e9-195b-4953-bcf4-e5f34f7949d9","Type":"ContainerStarted","Data":"ede8cdec8887fe556d49f10308ac82661dac89710c55b5242aa558b7e4297601"} Oct 29 22:30:34.115639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:34.115579 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podStartSLOduration=1.115563278 podStartE2EDuration="1.115563278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:34.113988453 +0000 UTC m=+896.595305947" watchObservedRunningTime="2026-10-29 22:30:34.115563278 +0000 UTC m=+896.596880772" Oct 29 22:30:34.385375 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:34.385276 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:34.387339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:34.387304 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:30:34.387507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:34.387356 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 29 22:30:34.536096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:34.536058 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:35.385850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:35.385810 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:30:35.386363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:35.385878 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 29 22:30:36.249637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:36.249607 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:36.392686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:36.392652 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:30:36.393231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:36.393202 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 29 22:30:36.404332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:36.404298 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:37.385810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:37.385772 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:30:37.386220 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:37.386176 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 29 22:30:38.097342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:38.097307 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:38.388466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:38.387799 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:30:38.388466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:38.387879 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 29 22:30:39.391417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:39.390932 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:30:39.391417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:39.391025 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 29 22:30:40.218877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:40.218838 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:40.385330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:40.385289 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 29 22:30:40.385548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:40.385357 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 29 22:30:41.328310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.328278 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc"] Oct 29 22:30:41.333912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.333890 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.347524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.347495 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"kube-root-ca.crt\"" Oct 29 22:30:41.357517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.357486 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"openshift-service-ca.crt\"" Oct 29 22:30:41.357681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.357555 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"distinct-w6-92fd86-kserve-self-signed-certs\"" Oct 29 22:30:41.357681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.357629 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-92fd86\"/\"default-dockercfg-f6qvf\"" Oct 29 22:30:41.360177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.360154 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc"] Oct 29 22:30:41.390869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.390840 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:41.391186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.391164 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:41.392020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.392001 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:30:41.424096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.424066 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:41.427486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.427462 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-tmp-dir\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.427614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.427498 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8052bf-194c-495c-a1d4-4431b30768ce-tls-certs\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.427614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.427578 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6gq\" (UniqueName: \"kubernetes.io/projected/2d8052bf-194c-495c-a1d4-4431b30768ce-kube-api-access-dd6gq\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.427759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.427738 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-home\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.428558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.428524 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-model-cache\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.428695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.428651 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-dshm\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.530326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.530286 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-tmp-dir\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.530326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.530329 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8052bf-194c-495c-a1d4-4431b30768ce-tls-certs\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.530602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.530360 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6gq\" (UniqueName: \"kubernetes.io/projected/2d8052bf-194c-495c-a1d4-4431b30768ce-kube-api-access-dd6gq\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.530602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.530405 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-home\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.530602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.530555 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-model-cache\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.530752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.530607 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-dshm\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.530802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.530747 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-tmp-dir\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.530929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.530907 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-model-cache\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.531103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.531088 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-home\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.532817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.532794 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-dshm\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.533227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.533206 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8052bf-194c-495c-a1d4-4431b30768ce-tls-certs\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.550368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.550337 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6gq\" (UniqueName: \"kubernetes.io/projected/2d8052bf-194c-495c-a1d4-4431b30768ce-kube-api-access-dd6gq\") pod \"distinct-w6-92fd86-kserve-9f776dd-gwphc\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.649543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.649433 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:41.813382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.808511 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:41.973184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:41.973148 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc"] Oct 29 22:30:41.982577 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:30:41.982549 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d8052bf_194c_495c_a1d4_4431b30768ce.slice/crio-09433dd988b324921d7ef3c864667f68016651ce6be788f727b3d3567f95edb0 WatchSource:0}: Error finding container 09433dd988b324921d7ef3c864667f68016651ce6be788f727b3d3567f95edb0: Status 404 returned error can't find the container with id 09433dd988b324921d7ef3c864667f68016651ce6be788f727b3d3567f95edb0 Oct 29 22:30:42.199990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:42.199955 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" event={"ID":"2d8052bf-194c-495c-a1d4-4431b30768ce","Type":"ContainerStarted","Data":"4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7"} Oct 29 22:30:42.200202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:42.200001 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" event={"ID":"2d8052bf-194c-495c-a1d4-4431b30768ce","Type":"ContainerStarted","Data":"09433dd988b324921d7ef3c864667f68016651ce6be788f727b3d3567f95edb0"} Oct 29 22:30:42.234923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:42.234829 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podStartSLOduration=1.234815611 podStartE2EDuration="1.234815611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:42.233945494 +0000 UTC m=+904.715262986" watchObservedRunningTime="2026-10-29 22:30:42.234815611 +0000 UTC m=+904.716133119" Oct 29 22:30:42.650563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:42.650486 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:43.346648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:43.346610 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:43.715485 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:30:43.715452 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:30:44.952610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:44.945490 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:46.460450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:46.460083 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:48.152615 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:48.152528 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:49.717742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.717703 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb"] Oct 29 22:30:49.722290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.722263 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.734993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.734908 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"kube-root-ca.crt\"" Oct 29 22:30:49.739059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.738978 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:49.745821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.745673 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"distinct-w2-8d3632-kserve-self-signed-certs\"" Oct 29 22:30:49.745821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.745722 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"openshift-service-ca.crt\"" Oct 29 22:30:49.745821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.745795 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"default-dockercfg-tlzc2\"" Oct 29 22:30:49.748614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.748595 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb"] Oct 29 22:30:49.818535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.818440 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-model-cache\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.818707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.818649 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-home\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.818767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.818732 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-dshm\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.818767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.818759 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5def10-271f-4669-896d-9183f5015de9-tls-certs\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.818869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.818848 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8th\" (UniqueName: \"kubernetes.io/projected/cf5def10-271f-4669-896d-9183f5015de9-kube-api-access-ml8th\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.818922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.818900 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-tmp-dir\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.874691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.874651 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.920529 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-tmp-dir\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.931550 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-model-cache\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.931823 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-home\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.931924 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-dshm\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.931953 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5def10-271f-4669-896d-9183f5015de9-tls-certs\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.932008 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8th\" (UniqueName: \"kubernetes.io/projected/cf5def10-271f-4669-896d-9183f5015de9-kube-api-access-ml8th\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.924001 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-tmp-dir\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.932809 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-model-cache\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.933423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.933102 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-home\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.952838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.952805 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8th\" (UniqueName: \"kubernetes.io/projected/cf5def10-271f-4669-896d-9183f5015de9-kube-api-access-ml8th\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.958057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.957503 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5def10-271f-4669-896d-9183f5015de9-tls-certs\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:49.999842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:49.999723 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-dshm\") pod \"distinct-w2-8d3632-kserve-5f6856f47-s9szb\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:50.046414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:50.038628 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:50.348526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:50.348455 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb"] Oct 29 22:30:50.355280 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:30:50.355254 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf5def10_271f_4669_896d_9183f5015de9.slice/crio-47d18f0f83da8c0db8eca5dc8252627d657061d3fa6b7baff3fc42ffc4a22629 WatchSource:0}: Error finding container 47d18f0f83da8c0db8eca5dc8252627d657061d3fa6b7baff3fc42ffc4a22629: Status 404 returned error can't find the container with id 47d18f0f83da8c0db8eca5dc8252627d657061d3fa6b7baff3fc42ffc4a22629 Oct 29 22:30:51.257749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:51.257714 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" event={"ID":"cf5def10-271f-4669-896d-9183f5015de9","Type":"ContainerStarted","Data":"7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2"} Oct 29 22:30:51.258149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:51.257759 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" event={"ID":"cf5def10-271f-4669-896d-9183f5015de9","Type":"ContainerStarted","Data":"47d18f0f83da8c0db8eca5dc8252627d657061d3fa6b7baff3fc42ffc4a22629"} Oct 29 22:30:51.295918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:51.295860 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podStartSLOduration=2.295843122 podStartE2EDuration="2.295843122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:30:51.291897877 +0000 UTC m=+913.773215372" watchObservedRunningTime="2026-10-29 22:30:51.295843122 +0000 UTC m=+913.777160616" Oct 29 22:30:51.529164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:51.529057 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:51.650147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:51.650117 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:51.664460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:51.664432 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:52.040358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:52.040325 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:30:52.275132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:52.275094 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:30:53.114141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:53.114106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:54.626415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:54.620976 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:56.416416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:56.415997 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:58.174278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:58.174243 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:30:59.614871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:30:59.614824 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:00.040103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:00.040061 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:31:00.052906 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:00.052876 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:31:00.334098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:00.334016 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:31:01.159284 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:01.159235 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:01.979351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:01.979319 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:02.801168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:02.801048 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:03.463174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:03.463138 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:04.497577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:04.497542 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:06.094219 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:06.094180 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:07.574410 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:07.574340 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:09.044751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:09.044710 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:10.625195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:10.625156 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:13.567895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:13.567860 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:13.713161 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:31:13.713120 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:31:15.113838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:15.113781 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:16.633384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:16.633344 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:17.632097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:17.632065 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:18.111005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.110967 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc"] Oct 29 22:31:18.117058 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.117036 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.145489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.145450 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc"] Oct 29 22:31:18.175244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175211 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4daf7ff6-9658-4276-90f2-a148d32143af-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.175425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175255 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.175425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175309 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.175425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175336 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.175425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175369 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.175679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175429 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.175679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175465 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2qw\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-kube-api-access-sx2qw\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.175679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175488 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.175679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.175503 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4daf7ff6-9658-4276-90f2-a148d32143af-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277138 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277181 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4daf7ff6-9658-4276-90f2-a148d32143af-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277271 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4daf7ff6-9658-4276-90f2-a148d32143af-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277316 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277453 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277513 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277567 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277596 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277623 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277680 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2qw\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-kube-api-access-sx2qw\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.277886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277861 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.278051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.277945 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.278162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.278049 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4daf7ff6-9658-4276-90f2-a148d32143af-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.278162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.278079 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.279710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.279694 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.279949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.279928 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4daf7ff6-9658-4276-90f2-a148d32143af-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.294730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.294706 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2qw\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-kube-api-access-sx2qw\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.294901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.294879 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.357241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.357146 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:18.439843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.439763 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:18.661061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.661025 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm"] Oct 29 22:31:18.665903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.665879 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.681566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.681538 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm"] Oct 29 22:31:18.769339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.769304 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc"] Oct 29 22:31:18.771140 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:31:18.771101 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4daf7ff6_9658_4276_90f2_a148d32143af.slice/crio-b1c7c6c5c88ed3e7a095cfb25c276a9af0840a7fe8e9f46ee8637c27fd3113c2 WatchSource:0}: Error finding container b1c7c6c5c88ed3e7a095cfb25c276a9af0840a7fe8e9f46ee8637c27fd3113c2: Status 404 returned error can't find the container with id b1c7c6c5c88ed3e7a095cfb25c276a9af0840a7fe8e9f46ee8637c27fd3113c2 Oct 29 22:31:18.773904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.773875 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:18.773997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.773936 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:18.773997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.773965 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:18.803273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803248 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.803367 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803292 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.803433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803377 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.803471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803436 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.803517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803497 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.803554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803540 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsmkm\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-kube-api-access-fsmkm\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.803649 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803626 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.803724 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803710 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.803786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.803775 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905054 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsmkm\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-kube-api-access-fsmkm\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905107 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905158 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905231 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905273 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905315 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905369 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905437 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905500 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905754 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905607 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905754 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905686 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.905754 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.905724 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.906045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.906021 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.906519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.906492 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.908218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.908188 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.908367 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.908347 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.925672 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.925644 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.925894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.925870 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsmkm\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-kube-api-access-fsmkm\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:18.941299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.941272 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9"] Oct 29 22:31:18.946590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.946568 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:18.968657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.968629 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9"] Oct 29 22:31:18.986414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:18.982630 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:19.006685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.006655 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.006896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.006876 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.007762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.007739 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.007944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.007922 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.008075 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.008060 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.008245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.008222 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqpb\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-kube-api-access-8xqpb\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.008344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.008298 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.008440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.008337 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924894a6-42ae-41e5-b760-71a6ea4ac86f-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.008440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.008416 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110274 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110326 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924894a6-42ae-41e5-b760-71a6ea4ac86f-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110381 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110449 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110548 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110625 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110673 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110713 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.110773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.110759 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqpb\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-kube-api-access-8xqpb\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.112011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.111988 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924894a6-42ae-41e5-b760-71a6ea4ac86f-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.112324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.112261 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.113705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.112870 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.113705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.113363 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.113705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.113647 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.115309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.115282 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.115518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.115500 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.127652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.127623 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqpb\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-kube-api-access-8xqpb\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.127773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.127752 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.263553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.263514 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:19.322541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.321992 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm"] Oct 29 22:31:19.346809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.346780 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:19.349541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.349510 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:19.349697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.349679 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:19.512942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.512902 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" event={"ID":"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75","Type":"ContainerStarted","Data":"aed7c9ad80c0295b60570733219bf4ea45cf2e5a6b55aad5380145fcf8dc1090"} Oct 29 22:31:19.514704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.514675 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" event={"ID":"4daf7ff6-9658-4276-90f2-a148d32143af","Type":"ContainerStarted","Data":"16db3dc1ec23bd55692695596d10f64a286cb3db6deb7dd965bcb878dd76f736"} Oct 29 22:31:19.514820 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.514726 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" event={"ID":"4daf7ff6-9658-4276-90f2-a148d32143af","Type":"ContainerStarted","Data":"b1c7c6c5c88ed3e7a095cfb25c276a9af0840a7fe8e9f46ee8637c27fd3113c2"} Oct 29 22:31:19.559147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.557218 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podStartSLOduration=1.5571952119999999 podStartE2EDuration="1.557195212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:19.555683138 +0000 UTC m=+942.037000631" watchObservedRunningTime="2026-10-29 22:31:19.557195212 +0000 UTC m=+942.038512706" Oct 29 22:31:19.607627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.607545 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9"] Oct 29 22:31:19.611713 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:31:19.611686 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924894a6_42ae_41e5_b760_71a6ea4ac86f.slice/crio-4fc088919d46a15cd3f18165bfbf6325966aac7fc857b7de4e7baeaa58e2fe1a WatchSource:0}: Error finding container 4fc088919d46a15cd3f18165bfbf6325966aac7fc857b7de4e7baeaa58e2fe1a: Status 404 returned error can't find the container with id 4fc088919d46a15cd3f18165bfbf6325966aac7fc857b7de4e7baeaa58e2fe1a Oct 29 22:31:19.614792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.614754 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:19.614915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.614831 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:19.614915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:19.614877 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:20.070906 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.070871 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:20.441164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.441076 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:20.442639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.442605 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:31:20.442816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.442797 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:31:20.550768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.550119 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" event={"ID":"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75","Type":"ContainerStarted","Data":"bd83f1aecd1335833096de025140e2d416598611f673d4adbf2893bcbc1aa605"} Oct 29 22:31:20.555442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.554336 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" event={"ID":"924894a6-42ae-41e5-b760-71a6ea4ac86f","Type":"ContainerStarted","Data":"eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef"} Oct 29 22:31:20.555442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.554387 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" event={"ID":"924894a6-42ae-41e5-b760-71a6ea4ac86f","Type":"ContainerStarted","Data":"4fc088919d46a15cd3f18165bfbf6325966aac7fc857b7de4e7baeaa58e2fe1a"} Oct 29 22:31:20.594417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.594120 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podStartSLOduration=2.594084305 podStartE2EDuration="2.594084305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:20.592213539 +0000 UTC m=+943.073531033" watchObservedRunningTime="2026-10-29 22:31:20.594084305 +0000 UTC m=+943.075401799" Oct 29 22:31:20.636416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.636170 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podStartSLOduration=2.636152004 podStartE2EDuration="2.636152004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:20.635953538 +0000 UTC m=+943.117271033" watchObservedRunningTime="2026-10-29 22:31:20.636152004 +0000 UTC m=+943.117469497" Oct 29 22:31:20.990416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.989788 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:21.000419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.993499 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:31:21.000419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:20.993562 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:31:21.266464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:21.265526 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:21.270529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:21.267338 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:31:21.270529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:21.267438 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:31:21.443416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:21.442827 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:31:21.443416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:21.442892 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:31:21.991794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:21.991757 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:31:21.992043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:21.992015 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:31:22.204416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:22.203913 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:22.268717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:22.268634 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:31:22.268983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:22.268954 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:31:22.449835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:22.449800 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:31:22.450416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:22.450366 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:31:22.985994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:22.985962 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:31:22.986285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:22.986257 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:31:23.269632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:23.269514 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:31:23.270291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:23.269856 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:31:23.458418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:23.446832 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:31:23.458418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:23.446908 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:31:23.989424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:23.988564 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:31:23.989424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:23.988621 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:31:24.278481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:24.266814 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:31:24.278481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:24.266880 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:31:24.452148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:24.452105 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:31:24.452481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:24.452446 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:31:24.473440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:24.472080 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:24.992290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:24.991822 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:31:24.992290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:24.991885 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:31:25.292483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:25.290253 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:31:25.292483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:25.290310 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:31:25.440214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:25.440170 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 29 22:31:25.440403 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:25.440234 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 29 22:31:25.987417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:25.986805 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:31:25.987417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:25.986869 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:31:26.264546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.264446 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 29 22:31:26.264813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.264772 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 29 22:31:26.558929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.558815 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:26.610729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.610691 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:26.612802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.612778 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:26.614950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.614929 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:31:26.797534 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.794420 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:26.986653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.986613 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 29 22:31:26.986846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:26.986678 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 29 22:31:27.434137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:27.434047 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:27.436418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:27.436381 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:27.438745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:27.438722 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:31:27.511811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:27.511772 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:27.990317 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:27.990291 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:27.990677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:27.990658 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:27.991459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:27.991440 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:31:28.223168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:28.223134 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:29.758800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:29.758765 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:29.997874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:29.997835 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:32.710418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:32.710172 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:33.431418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.421495 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh"] Oct 29 22:31:33.445785 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.445755 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.485461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.475439 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh"] Oct 29 22:31:33.556178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.556133 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.556467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.556441 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.556749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.556730 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgf2j\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-kube-api-access-xgf2j\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.556964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.556947 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.557122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.557097 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.557249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.557234 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.557356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.557342 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.561353 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.561329 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.561554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.561531 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17df44f9-f686-482d-a4f4-49bb0ba40541-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.664189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.664152 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.664490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.664448 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.664714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.664677 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.664898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.664879 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.665075 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.665058 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.665302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.665285 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17df44f9-f686-482d-a4f4-49bb0ba40541-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.665494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.665475 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.665649 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.665634 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.665886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.665862 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgf2j\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-kube-api-access-xgf2j\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.666773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.666751 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.667211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.667188 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.667646 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.667624 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.668786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.668745 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17df44f9-f686-482d-a4f4-49bb0ba40541-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.669574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.669553 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.696547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.696081 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.709695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.707720 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.709695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.708353 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgf2j\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-kube-api-access-xgf2j\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.709695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.708769 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.852571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.852535 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2"] Oct 29 22:31:33.858595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.858573 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.880434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.880405 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2"] Oct 29 22:31:33.909247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.909211 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970320 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970496 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83986232-6de2-4f0c-84f4-527002396e43-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970658 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970710 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83986232-6de2-4f0c-84f4-527002396e43-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970746 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970830 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970879 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970922 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm65h\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-kube-api-access-mm65h\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:33.971218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:33.970981 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.073216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073170 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.073357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073224 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.073357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073255 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm65h\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-kube-api-access-mm65h\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.073357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073317 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.073483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073460 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.075924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073531 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83986232-6de2-4f0c-84f4-527002396e43-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.075924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073646 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.075924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073697 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83986232-6de2-4f0c-84f4-527002396e43-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.075924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.073730 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.075924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.074194 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.075924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.074449 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.075924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.075205 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.075924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.075826 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.077375 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.076523 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83986232-6de2-4f0c-84f4-527002396e43-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.077976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.077953 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.079325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.079299 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83986232-6de2-4f0c-84f4-527002396e43-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.099953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.099927 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.101585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.101551 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm65h\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-kube-api-access-mm65h\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.175802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.175771 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:34.265821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.265769 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh"] Oct 29 22:31:34.266996 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:31:34.266957 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17df44f9_f686_482d_a4f4_49bb0ba40541.slice/crio-9353660c0d960411ea786089ebae25e11047a17c373b8cad97a3d436d3fb1724 WatchSource:0}: Error finding container 9353660c0d960411ea786089ebae25e11047a17c373b8cad97a3d436d3fb1724: Status 404 returned error can't find the container with id 9353660c0d960411ea786089ebae25e11047a17c373b8cad97a3d436d3fb1724 Oct 29 22:31:34.269808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.269775 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:34.269923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.269837 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:34.269923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.269866 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:34.482111 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.481206 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2"] Oct 29 22:31:34.484530 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:31:34.484500 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83986232_6de2_4f0c_84f4_527002396e43.slice/crio-50e2141cca7776477c9c779932b880902f5b0b4db01f00e20479aec557c629f9 WatchSource:0}: Error finding container 50e2141cca7776477c9c779932b880902f5b0b4db01f00e20479aec557c629f9: Status 404 returned error can't find the container with id 50e2141cca7776477c9c779932b880902f5b0b4db01f00e20479aec557c629f9 Oct 29 22:31:34.487308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.487273 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:34.487433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.487362 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:34.487433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.487421 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:34.631025 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.630929 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:34.991976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.991934 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" event={"ID":"83986232-6de2-4f0c-84f4-527002396e43","Type":"ContainerStarted","Data":"6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29"} Oct 29 22:31:34.992508 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.992010 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" event={"ID":"83986232-6de2-4f0c-84f4-527002396e43","Type":"ContainerStarted","Data":"50e2141cca7776477c9c779932b880902f5b0b4db01f00e20479aec557c629f9"} Oct 29 22:31:34.993501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.993469 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" event={"ID":"17df44f9-f686-482d-a4f4-49bb0ba40541","Type":"ContainerStarted","Data":"3779faeb6aeaeed935f68dde5deef4ee1ea3930f9c884353fb2c7ebba7bfc833"} Oct 29 22:31:34.993602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:34.993514 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" event={"ID":"17df44f9-f686-482d-a4f4-49bb0ba40541","Type":"ContainerStarted","Data":"9353660c0d960411ea786089ebae25e11047a17c373b8cad97a3d436d3fb1724"} Oct 29 22:31:35.035209 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:35.035154 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podStartSLOduration=2.035140676 podStartE2EDuration="2.035140676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:35.033131003 +0000 UTC m=+957.514448502" watchObservedRunningTime="2026-10-29 22:31:35.035140676 +0000 UTC m=+957.516458169" Oct 29 22:31:35.083412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:35.076920 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podStartSLOduration=2.076879425 podStartE2EDuration="2.076879425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:35.072445436 +0000 UTC m=+957.553762930" watchObservedRunningTime="2026-10-29 22:31:35.076879425 +0000 UTC m=+957.558196918" Oct 29 22:31:35.178417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:35.177762 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:35.186420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:35.183509 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:31:35.186420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:35.183571 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 29 22:31:35.926551 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:35.926511 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:35.928190 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:35.928167 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:31:35.928266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:35.928215 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:31:36.181483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:36.180827 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:31:36.181483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:36.180890 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 29 22:31:36.420417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:36.419498 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:36.574417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:36.569072 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:36.926409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:36.926309 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:31:36.926409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:36.926361 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:31:37.191476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:37.188819 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:31:37.191476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:37.188880 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 29 22:31:37.925813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:37.925779 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:31:37.926049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:37.925831 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:31:38.186485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:38.177803 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:31:38.186485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:38.177866 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 29 22:31:38.470630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:38.470592 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:38.926651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:38.926568 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:31:38.926651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:38.926636 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:31:39.181656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:39.181569 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:31:39.181886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:39.181861 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 29 22:31:39.927004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:39.926960 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:31:39.927647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:39.927600 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:31:40.173071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:40.173009 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:40.177825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:40.177751 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:31:40.178055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:40.178029 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 29 22:31:40.498342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:40.498302 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:40.938537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:40.933187 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 29 22:31:40.940728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:40.938769 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 29 22:31:41.177174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:41.176739 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 29 22:31:41.177174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:41.176804 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 29 22:31:41.931479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:41.931441 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:41.932099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:41.932046 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:41.932989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:41.932967 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:31:41.958101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:41.958069 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:42.345874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:42.345841 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:42.346939 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:42.346924 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:42.347713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:42.347688 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:31:43.646613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:43.646030 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:43.773387 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:31:43.773348 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:31:45.013697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.013652 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8"] Oct 29 22:31:45.019637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.019592 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.038213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.038185 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8"] Oct 29 22:31:45.045869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.045845 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8d3632\"/\"distinct-2-w2-8d3632-kserve-self-signed-certs\"" Oct 29 22:31:45.078009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.077976 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-dshm\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.078226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.078196 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-model-cache\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.078301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.078264 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7424bca-507a-4c13-a526-a67de510814e-tls-certs\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.078512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.078487 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-tmp-dir\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.078564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.078541 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-home\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.078611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.078589 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spm7v\" (UniqueName: \"kubernetes.io/projected/b7424bca-507a-4c13-a526-a67de510814e-kube-api-access-spm7v\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.137548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.137511 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:45.180035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180007 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-dshm\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.180035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180038 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-model-cache\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.180273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180078 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7424bca-507a-4c13-a526-a67de510814e-tls-certs\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.180273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180186 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-tmp-dir\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.180273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180241 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-home\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.180634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180350 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spm7v\" (UniqueName: \"kubernetes.io/projected/b7424bca-507a-4c13-a526-a67de510814e-kube-api-access-spm7v\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.180985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180654 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-model-cache\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.181082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180704 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-home\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.181082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.180763 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-tmp-dir\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.182794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.182764 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7424bca-507a-4c13-a526-a67de510814e-tls-certs\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.182886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.182766 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-dshm\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.199013 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.198975 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spm7v\" (UniqueName: \"kubernetes.io/projected/b7424bca-507a-4c13-a526-a67de510814e-kube-api-access-spm7v\") pod \"distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:45.286037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.285902 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:45.358086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:45.358053 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:46.051933 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:31:46.051873 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7424bca_507a_4c13_a526_a67de510814e.slice/crio-6c0ad7690225e22000ff8f061d4e6a8f72af25a02f9e8c78cf3d6bc9edd89183 WatchSource:0}: Error finding container 6c0ad7690225e22000ff8f061d4e6a8f72af25a02f9e8c78cf3d6bc9edd89183: Status 404 returned error can't find the container with id 6c0ad7690225e22000ff8f061d4e6a8f72af25a02f9e8c78cf3d6bc9edd89183 Oct 29 22:31:46.053301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:46.053073 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8"] Oct 29 22:31:46.151837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:46.151773 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" event={"ID":"b7424bca-507a-4c13-a526-a67de510814e","Type":"ContainerStarted","Data":"6c0ad7690225e22000ff8f061d4e6a8f72af25a02f9e8c78cf3d6bc9edd89183"} Oct 29 22:31:47.185532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:47.185487 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:47.209417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:47.201715 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" event={"ID":"b7424bca-507a-4c13-a526-a67de510814e","Type":"ContainerStarted","Data":"7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f"} Oct 29 22:31:47.247266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:47.247192 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podStartSLOduration=3.247171949 podStartE2EDuration="3.247171949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:47.243600266 +0000 UTC m=+969.724917778" watchObservedRunningTime="2026-10-29 22:31:47.247171949 +0000 UTC m=+969.728489443" Oct 29 22:31:47.323919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:47.310490 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:47.334413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:47.326088 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:31:47.361998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:47.361963 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:48.269425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.268739 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97"] Oct 29 22:31:48.283415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.274102 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.335916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.335877 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97"] Oct 29 22:31:48.454599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.454559 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884ab263-fded-4dec-b6f5-bb9799fbb140-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.454886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.454861 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.455020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.454898 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.455020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.454980 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.455118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.455105 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.455205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.455188 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.455305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.455290 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtwr\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-kube-api-access-nwtwr\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.455454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.455383 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.455565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.455547 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.492984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.492952 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.559843 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884ab263-fded-4dec-b6f5-bb9799fbb140-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.559940 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.559962 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.559998 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.560047 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.560102 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.560194 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtwr\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-kube-api-access-nwtwr\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.560223 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.560318 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.563375 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884ab263-fded-4dec-b6f5-bb9799fbb140-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.563581 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.563658 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.564490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.563934 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.565191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.564505 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.565191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.564741 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.568422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.567015 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.585360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.585324 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtwr\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-kube-api-access-nwtwr\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.596435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.593830 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.626830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.626795 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:48.756935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.756897 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9"] Oct 29 22:31:48.762299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.762274 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.788636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.788603 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9"] Oct 29 22:31:48.864218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864138 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.864218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864176 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.864425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864309 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.864425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864359 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.864493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864431 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgfrw\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-kube-api-access-bgfrw\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.864493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864458 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.864493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864483 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.864752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864726 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1d4d6d-d183-4e43-8179-d0e57be42478-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.864788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.864769 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.948335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.948307 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97"] Oct 29 22:31:48.949767 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:31:48.949735 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884ab263_fded_4dec_b6f5_bb9799fbb140.slice/crio-dfee91db9eb2f17be8d4e9137e5e56085e1849aa113055600ea899e98aec4977 WatchSource:0}: Error finding container dfee91db9eb2f17be8d4e9137e5e56085e1849aa113055600ea899e98aec4977: Status 404 returned error can't find the container with id dfee91db9eb2f17be8d4e9137e5e56085e1849aa113055600ea899e98aec4977 Oct 29 22:31:48.952431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.952384 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:48.952528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.952466 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:48.952528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.952501 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:48.965949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.965913 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966006 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966053 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966094 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgfrw\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-kube-api-access-bgfrw\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966137 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966171 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966320 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-data\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966338 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-certs\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966340 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1d4d6d-d183-4e43-8179-d0e57be42478-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966407 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966385 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966493 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966712 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-credential-socket\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.966986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.966967 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1d4d6d-d183-4e43-8179-d0e57be42478-istiod-ca-cert\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.968909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.968884 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-envoy\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.969101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.969084 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-podinfo\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.984971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.984952 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-token\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:48.985130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:48.985110 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgfrw\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-kube-api-access-bgfrw\") pod \"e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:49.086866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.086821 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:49.244148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.244109 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" event={"ID":"884ab263-fded-4dec-b6f5-bb9799fbb140","Type":"ContainerStarted","Data":"c97f5a6f53159efd9f0210794f0704b4bd7a27fe5aa9e5293d8bfd2af8ac2c9b"} Oct 29 22:31:49.244445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.244380 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" event={"ID":"884ab263-fded-4dec-b6f5-bb9799fbb140","Type":"ContainerStarted","Data":"dfee91db9eb2f17be8d4e9137e5e56085e1849aa113055600ea899e98aec4977"} Oct 29 22:31:49.287732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.287674 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podStartSLOduration=1.2876550039999999 podStartE2EDuration="1.287655004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:49.285177182 +0000 UTC m=+971.766494676" watchObservedRunningTime="2026-10-29 22:31:49.287655004 +0000 UTC m=+971.768972497" Oct 29 22:31:49.406642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.406421 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9"] Oct 29 22:31:49.409009 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:31:49.408962 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1d4d6d_d183_4e43_8179_d0e57be42478.slice/crio-c6fe159b4ab1a39c13c56cefd4f04f86b8d350126e89b775b738beb9d2faf110 WatchSource:0}: Error finding container c6fe159b4ab1a39c13c56cefd4f04f86b8d350126e89b775b738beb9d2faf110: Status 404 returned error can't find the container with id c6fe159b4ab1a39c13c56cefd4f04f86b8d350126e89b775b738beb9d2faf110 Oct 29 22:31:49.412512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.412480 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:49.412608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.412548 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:49.412608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.412578 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:31:49.636784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:49.636754 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:50.150248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:50.150212 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:50.362418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:50.359108 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" event={"ID":"3f1d4d6d-d183-4e43-8179-d0e57be42478","Type":"ContainerStarted","Data":"37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428"} Oct 29 22:31:50.362418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:50.359207 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" event={"ID":"3f1d4d6d-d183-4e43-8179-d0e57be42478","Type":"ContainerStarted","Data":"c6fe159b4ab1a39c13c56cefd4f04f86b8d350126e89b775b738beb9d2faf110"} Oct 29 22:31:50.445458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:50.445384 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podStartSLOduration=2.445362045 podStartE2EDuration="2.445362045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:31:50.429940695 +0000 UTC m=+972.911258201" watchObservedRunningTime="2026-10-29 22:31:50.445362045 +0000 UTC m=+972.926679540" Oct 29 22:31:50.630732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:50.630690 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:50.630902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:50.630752 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 29 22:31:51.095417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:51.091546 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:51.095417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:51.091709 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:51.095417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:51.091754 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 29 22:31:51.628018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:51.627987 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:51.628476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:51.628450 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 29 22:31:52.067772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:52.067723 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:52.088562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:52.088529 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:52.088852 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:52.088807 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 29 22:31:52.627791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:52.627755 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:52.627995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:52.627815 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 29 22:31:53.088848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:53.088812 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:53.089828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:53.089426 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 29 22:31:53.629421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:53.627793 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:53.629421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:53.627858 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 29 22:31:53.888152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:53.888041 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:54.089428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:54.089372 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:54.090945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:54.090479 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 29 22:31:54.436456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:54.432711 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:54.660419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:54.657735 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:54.660419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:54.657794 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 29 22:31:55.093416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:55.087808 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:55.093416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:55.087870 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 29 22:31:55.358786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:55.358697 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:55.408924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:55.408896 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:55.504153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:55.504119 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:31:55.628242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:55.628143 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 29 22:31:55.629146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:55.629113 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 29 22:31:56.001837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:56.001782 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:56.102418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:56.099658 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:31:56.102418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:56.099738 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 29 22:31:56.631613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:56.631583 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:56.632000 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:56.631978 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:56.632652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:56.632629 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:31:57.342858 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:57.342827 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:57.343793 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:57.343766 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:57.344783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:57.344752 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:31:57.643625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:57.643547 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:31:59.156842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:31:59.156814 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:00.818448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:00.816624 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:03.267565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:03.266278 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:32:03.329282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:03.328519 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.504164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:03.503544 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:03.698494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:03.697905 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff"] Oct 29 22:32:03.830422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:03.829450 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd"] Oct 29 22:32:03.834594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:03.834555 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:03.968469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:03.947157 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.038417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:03.983052 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.038860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.038822 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfj4g\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-kube-api-access-vfj4g\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.055451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.046502 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.075420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.059445 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.075695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.075669 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.120714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.075917 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.140846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.140707 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.141313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.141290 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.141592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.141571 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.155586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.090235 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:04.170566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.101518 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff"] Oct 29 22:32:04.170566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.144544 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:04.270035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.269927 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:04.270859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.270815 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.271173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.271153 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.271472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.271341 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.271660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.271628 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6pj6\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-kube-api-access-v6pj6\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.271880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.271860 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7635edde-2274-444f-a076-b39842f07f12-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.297756 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.272719 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:32:04.298062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.273914 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.298240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.295891 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.298454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.297607 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.298692 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.298676 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfj4g\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-kube-api-access-vfj4g\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.298864 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.298844 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.299332 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309503 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309566 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309592 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7635edde-2274-444f-a076-b39842f07f12-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309684 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309710 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309758 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309850 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309881 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.309938 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.310337 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.311435 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd"] Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.311802 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.312094 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.329371 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.329870 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.336416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.331715 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:32:04.464657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.464588 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfj4g\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-kube-api-access-vfj4g\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.496917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.493940 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.518678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.514432 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536104 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536312 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536568 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6pj6\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-kube-api-access-v6pj6\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536601 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7635edde-2274-444f-a076-b39842f07f12-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536664 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536710 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536734 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536789 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.537496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.536847 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7635edde-2274-444f-a076-b39842f07f12-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.549356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.541022 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7635edde-2274-444f-a076-b39842f07f12-istio-podinfo\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.549356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.541336 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-certs\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.549356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.545514 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7635edde-2274-444f-a076-b39842f07f12-istiod-ca-cert\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.549356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.546077 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.549356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.546360 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-credential-socket\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.549356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.549027 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.549356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.549281 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-data\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.587014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.586979 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:32:04.610422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.605573 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-envoy\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.613728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.610260 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:32:04.716417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.708442 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6pj6\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-kube-api-access-v6pj6\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.716417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.711488 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:32:04.745418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.718030 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:04.745418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.734878 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-istio-token\") pod \"e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.745418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.741238 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:04.949558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:04.944013 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:32:05.191123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.191088 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd"] Oct 29 22:32:05.191910 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:32:05.191885 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7635edde_2274_444f_a076_b39842f07f12.slice/crio-f10bf86e1581df5833851c5855c759611c0cbe75ccd3e67e09a7c112b1e2896d WatchSource:0}: Error finding container f10bf86e1581df5833851c5855c759611c0cbe75ccd3e67e09a7c112b1e2896d: Status 404 returned error can't find the container with id f10bf86e1581df5833851c5855c759611c0cbe75ccd3e67e09a7c112b1e2896d Oct 29 22:32:05.194540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.194500 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:05.194625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.194588 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:05.194665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.194632 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:05.201028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.200951 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff"] Oct 29 22:32:05.212848 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:32:05.212789 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdcd015_f56a_4206_b8cb_d9fe6cfbb68a.slice/crio-eceac7cb69b94c6dfbf53b6f5195b45ec477fc45371928c28836f4f769c75a81 WatchSource:0}: Error finding container eceac7cb69b94c6dfbf53b6f5195b45ec477fc45371928c28836f4f769c75a81: Status 404 returned error can't find the container with id eceac7cb69b94c6dfbf53b6f5195b45ec477fc45371928c28836f4f769c75a81 Oct 29 22:32:05.216072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.216038 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:05.216147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.216118 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:05.216182 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.216161 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:05.655467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.653088 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" event={"ID":"7635edde-2274-444f-a076-b39842f07f12","Type":"ContainerStarted","Data":"8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b"} Oct 29 22:32:05.655467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.653158 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" event={"ID":"7635edde-2274-444f-a076-b39842f07f12","Type":"ContainerStarted","Data":"f10bf86e1581df5833851c5855c759611c0cbe75ccd3e67e09a7c112b1e2896d"} Oct 29 22:32:05.655467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.654930 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" event={"ID":"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a","Type":"ContainerStarted","Data":"bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241"} Oct 29 22:32:05.655467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.654965 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" event={"ID":"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a","Type":"ContainerStarted","Data":"eceac7cb69b94c6dfbf53b6f5195b45ec477fc45371928c28836f4f769c75a81"} Oct 29 22:32:05.719893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.719839 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podStartSLOduration=2.7198223759999998 podStartE2EDuration="2.719822376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:05.71685911 +0000 UTC m=+988.198176823" watchObservedRunningTime="2026-10-29 22:32:05.719822376 +0000 UTC m=+988.201139872" Oct 29 22:32:05.729099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.729065 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:05.746156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.746129 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:05.795170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:05.795109 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podStartSLOduration=2.7950909729999998 podStartE2EDuration="2.795090973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:05.79408577 +0000 UTC m=+988.275403267" watchObservedRunningTime="2026-10-29 22:32:05.795090973 +0000 UTC m=+988.276408474" Oct 29 22:32:06.040411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:06.037051 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:06.735556 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:06.735510 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:32:06.736010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:06.735589 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 29 22:32:06.746904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:06.746876 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:32:06.747039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:06.746922 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 22:32:07.729638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:07.729526 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:32:07.729638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:07.729583 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 29 22:32:07.746312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:07.746262 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:32:07.746738 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:07.746330 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 22:32:08.076467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:08.075364 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:08.730229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:08.730181 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:32:08.730452 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:08.730255 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 29 22:32:08.746643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:08.746499 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:32:08.746643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:08.746554 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 22:32:09.730067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:09.730027 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:32:09.730298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:09.730091 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 29 22:32:09.746811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:09.746784 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:32:09.747171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:09.746830 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 22:32:09.899116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:09.899081 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:10.729745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:10.729706 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:32:10.729955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:10.729771 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 29 22:32:10.746192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:10.746159 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:32:10.746343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:10.746203 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 22:32:11.729256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:11.729220 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:32:11.730054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:11.730017 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 29 22:32:11.746521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:11.746490 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:32:11.746774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:11.746746 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 22:32:11.867907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:11.867868 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:12.729624 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:12.729588 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 29 22:32:12.730085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:12.729642 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 29 22:32:12.745919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:12.745886 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 29 22:32:12.746069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:12.745946 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 29 22:32:13.716571 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:32:13.716516 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:32:13.737184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:13.737145 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:13.737663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:13.737585 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:13.738444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:13.738414 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 22:32:13.753039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:13.751792 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:13.753039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:13.752714 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:13.753414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:13.753374 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:32:13.852634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:13.852601 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:14.064027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:14.063911 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:15.434640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:15.434603 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:17.027939 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:17.027889 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:18.415269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.415229 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx"] Oct 29 22:32:18.421159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.421140 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.450511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.450481 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx"] Oct 29 22:32:18.461453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.457027 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b"] Oct 29 22:32:18.477720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.477690 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.502469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.502440 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b"] Oct 29 22:32:18.512217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512189 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.512432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512229 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.512432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512270 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.512632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512462 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.512632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512508 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.512632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512546 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.512632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512577 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.512632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512600 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nc2v\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-kube-api-access-5nc2v\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.512840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512681 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.512840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512722 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.512840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512754 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.512840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512786 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa15275f-8808-4e9a-bb73-875dedce34c6-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.513077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512846 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.513077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.512902 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2dr7\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-kube-api-access-m2dr7\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.513077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.513014 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.513077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.513052 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.513349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.513089 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.513349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.513135 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.614129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614095 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.614312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614147 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2dr7\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-kube-api-access-m2dr7\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.614312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614207 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.614312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614241 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.614312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614276 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.614564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614320 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.614564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614371 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.614564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614415 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.614564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614457 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.614770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614573 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.614770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614606 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.614770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614637 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.614770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614669 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.614770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614688 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.615049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614694 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nc2v\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-kube-api-access-5nc2v\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.615049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614938 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.615049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614967 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.615049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.614992 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.615049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615034 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.615314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615064 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa15275f-8808-4e9a-bb73-875dedce34c6-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.615314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615187 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-credential-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.615314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615236 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.615314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615267 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-certs\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.615552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615525 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.615725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615705 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-socket\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.615863 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615840 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa15275f-8808-4e9a-bb73-875dedce34c6-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.616005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.615984 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istiod-ca-cert\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.616236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.616215 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-data\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.617180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.617159 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.617842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.617820 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.618703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.618674 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-envoy\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.619282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.619264 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-podinfo\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.633968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.633940 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.634109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.634094 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-token\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.634571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.634548 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nc2v\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-kube-api-access-5nc2v\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.635138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.635115 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2dr7\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-kube-api-access-m2dr7\") pod \"e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.737337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.737023 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:18.860303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.858547 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:18.888642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:18.886010 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:19.110011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.109975 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt"] Oct 29 22:32:19.116442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.116421 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.140712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.140675 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt"] Oct 29 22:32:19.150463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.150293 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c"] Oct 29 22:32:19.155436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.155372 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.191966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.191932 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c"] Oct 29 22:32:19.218029 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:32:19.216795 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa15275f_8808_4e9a_bb73_875dedce34c6.slice/crio-bd5239840eb1d16e6b90a0640839e071ab07ce73d067dd3406c01b2102f24e75 WatchSource:0}: Error finding container bd5239840eb1d16e6b90a0640839e071ab07ce73d067dd3406c01b2102f24e75: Status 404 returned error can't find the container with id bd5239840eb1d16e6b90a0640839e071ab07ce73d067dd3406c01b2102f24e75 Oct 29 22:32:19.220838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.220806 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.221030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.221005 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.221178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.221158 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.223184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.223167 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx"] Oct 29 22:32:19.226874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.225156 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.226874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.225228 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.226874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.225272 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.227537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.227517 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b"] Oct 29 22:32:19.283263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283185 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.283412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283292 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.283412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283320 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.283412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283356 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.283623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283447 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.283623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283581 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.283623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283611 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clq5n\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-kube-api-access-clq5n\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.283843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283677 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjt9\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-kube-api-access-4cjt9\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.283843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283728 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.283843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283746 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.284045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283872 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.284045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283935 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52788e-737b-4c5b-ac84-22329e92373a-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.284045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.283997 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.284254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.284061 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d78288-6f8f-4815-a748-ba548bd1d97e-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.284254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.284095 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52788e-737b-4c5b-ac84-22329e92373a-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.284254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.284112 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.284254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.284172 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.284254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.284195 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.385944 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.385993 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386016 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386049 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386087 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386127 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386153 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clq5n\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-kube-api-access-clq5n\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386190 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjt9\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-kube-api-access-4cjt9\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386257 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386288 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386363 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386444 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52788e-737b-4c5b-ac84-22329e92373a-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386523 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386591 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d78288-6f8f-4815-a748-ba548bd1d97e-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386633 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52788e-737b-4c5b-ac84-22329e92373a-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386670 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386770 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.386812 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.387703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.387310 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.389679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.389649 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.390346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.390298 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.390995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.390940 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d78288-6f8f-4815-a748-ba548bd1d97e-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.391316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.391250 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.393477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.392163 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-data\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.393477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.392763 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.393477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.392943 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.393477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.393172 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-credential-socket\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.393747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.393630 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-certs\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.394027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.393996 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52788e-737b-4c5b-ac84-22329e92373a-istiod-ca-cert\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.394890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.394868 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-envoy\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.395521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.395465 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.397344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.397321 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52788e-737b-4c5b-ac84-22329e92373a-istio-podinfo\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.408154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.407676 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.409446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.409411 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-istio-token\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.411633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.411013 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clq5n\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-kube-api-access-clq5n\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.411633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.411589 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjt9\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-kube-api-access-4cjt9\") pod \"e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.434037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.433757 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:19.507749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.504190 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:19.784035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.772503 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt"] Oct 29 22:32:19.784035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.781582 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.784035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.781681 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.784035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.781730 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.862248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.862199 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c"] Oct 29 22:32:19.874346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.874310 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.874795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.874774 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.874937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.874924 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:19.949628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.948548 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" event={"ID":"aa15275f-8808-4e9a-bb73-875dedce34c6","Type":"ContainerStarted","Data":"bdd0e9146b6f091fd3efa903052377e1da705dd6981cfa93da504f47803949a4"} Oct 29 22:32:19.949628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.948612 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" event={"ID":"aa15275f-8808-4e9a-bb73-875dedce34c6","Type":"ContainerStarted","Data":"bd5239840eb1d16e6b90a0640839e071ab07ce73d067dd3406c01b2102f24e75"} Oct 29 22:32:19.951992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.951958 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" event={"ID":"6b52788e-737b-4c5b-ac84-22329e92373a","Type":"ContainerStarted","Data":"b9306da14f2c37e0a257dcdb9db12878dcd77f16434c28de5bb66f7217b815f3"} Oct 29 22:32:19.954302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.954251 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" event={"ID":"95d78288-6f8f-4815-a748-ba548bd1d97e","Type":"ContainerStarted","Data":"ed1d898e6cd5a69ff55d1b1389e841b64f2351d217f85d1f30f06386cb23b85b"} Oct 29 22:32:19.956169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.956147 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" event={"ID":"d7084709-6ae5-4bba-b2b1-6c590b3ee21d","Type":"ContainerStarted","Data":"04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463"} Oct 29 22:32:19.956336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:19.956318 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" event={"ID":"d7084709-6ae5-4bba-b2b1-6c590b3ee21d","Type":"ContainerStarted","Data":"3688b380303b7fdc00674bbecda26c92f98c9275c46dd5cda863d6f9a29edac7"} Oct 29 22:32:20.023563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.018963 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podStartSLOduration=2.018944997 podStartE2EDuration="2.018944997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:20.009037537 +0000 UTC m=+1002.490355031" watchObservedRunningTime="2026-10-29 22:32:20.018944997 +0000 UTC m=+1002.500262491" Oct 29 22:32:20.057836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.057774 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podStartSLOduration=2.057754951 podStartE2EDuration="2.057754951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:20.051106789 +0000 UTC m=+1002.532424283" watchObservedRunningTime="2026-10-29 22:32:20.057754951 +0000 UTC m=+1002.539072446" Oct 29 22:32:20.740694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.740665 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:20.742568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.742542 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:20.742647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.742599 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:20.871448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.867275 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:20.871448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.867418 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:20.871448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.867463 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:20.950432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:20.937478 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:21.031592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.030354 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" event={"ID":"95d78288-6f8f-4815-a748-ba548bd1d97e","Type":"ContainerStarted","Data":"f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1"} Oct 29 22:32:21.051182 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.051146 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" event={"ID":"6b52788e-737b-4c5b-ac84-22329e92373a","Type":"ContainerStarted","Data":"bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644"} Oct 29 22:32:21.204195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.204121 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podStartSLOduration=2.204098471 podStartE2EDuration="2.204098471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:21.082995583 +0000 UTC m=+1003.564313081" watchObservedRunningTime="2026-10-29 22:32:21.204098471 +0000 UTC m=+1003.685415965" Oct 29 22:32:21.234479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.220955 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podStartSLOduration=2.220932958 podStartE2EDuration="2.220932958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:21.152352955 +0000 UTC m=+1003.633670451" watchObservedRunningTime="2026-10-29 22:32:21.220932958 +0000 UTC m=+1003.702250452" Oct 29 22:32:21.446474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.436665 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:21.446474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.436839 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.446474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.436896 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:21.519240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.519205 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:21.525005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.524965 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.525237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.525192 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:21.762484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.755406 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.762484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.755465 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:21.864437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.864380 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:21.864763 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.864708 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:21.955543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:21.955502 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:22.439426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.435111 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.439754 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.439714 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:22.520420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.518081 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.520420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.518155 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:22.734417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.733495 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:32:22.740770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.740737 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.740982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.740958 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:22.870627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.870593 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:22.871196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:22.871166 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:23.002471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.001532 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.002471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.001867 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:32:23.358918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.338775 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:23.414429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.413579 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:23.537711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.496495 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.537711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.496558 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:23.537711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.514744 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.537711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.514797 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:23.538056 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.537983 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:32:23.578650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.568983 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:23.900424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.891442 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.900424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.891504 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:23.934214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.934182 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:23.934523 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:23.934481 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:24.474904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.474871 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.475168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.475143 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:24.514720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.514687 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.515172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.515138 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:24.541523 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.541485 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:24.567807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.563634 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:32:24.855036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.854937 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.889760 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.889715 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:24.949419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.941249 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:24.949419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:24.941306 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:25.361417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.360794 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:32:25.485418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.436297 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.485418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.436372 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:25.545639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.545592 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:32:25.637555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.637453 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:32:25.913480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.913188 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.913480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.913257 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:25.954416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.933032 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:32:25.959829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.959785 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:25.961545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:25.961485 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:26.063152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.063120 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.063455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.063423 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:26.470453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.358103 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.561764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.526258 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.561764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.561649 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:26.561764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.559996 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.561764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.561703 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:26.561764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.550560 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.740577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.740489 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.772541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.772481 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:26.852579 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.852534 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.853195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.853172 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:32:26.891036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.890947 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.891348 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.891323 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:32:26.956429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.954268 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:26.957034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.956982 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:26.974779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.974724 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:26.980425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:26.979366 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:32:27.066445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.066211 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:27.493416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.461961 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:27.493416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.462016 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:27.592418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.531659 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:27.596535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.596495 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:28.004427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.979820 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:32:28.137833 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.852907 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:28.138823 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.876747 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:32:28.140198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.781540 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:28.140371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.140346 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:28.259366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:27.873697 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:28.260899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.260853 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:28.264607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.177545 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:32:28.549659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.546167 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:28.549659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.546241 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:28.576661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.576606 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:28.721982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.721946 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:28.723847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.723813 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:28.748439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.748387 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:28.748689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.748660 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:28.873619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.873532 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:28.873873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.873843 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:28.882618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:28.882569 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:29.139527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.139197 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:29.244203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.244161 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:32:29.432501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.432369 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:29.480001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.479966 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:29.480286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.480245 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:29.641087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.588845 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 29 22:32:29.641087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.588904 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 29 22:32:29.740315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.740282 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 29 22:32:29.740607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.740576 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 29 22:32:29.883031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.882957 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 29 22:32:29.883305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:29.883272 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 29 22:32:30.436560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:30.436074 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 29 22:32:30.436560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:30.436133 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 29 22:32:31.022088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.022044 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:31.026180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.026159 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:31.028041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.027886 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:32:31.050425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.050177 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:31.558920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.558888 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:31.575589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.565818 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:32:31.575589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.574492 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:31.575589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.574951 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:31.623237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.623191 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:32:31.623680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.623659 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:31.648037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.647992 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:31.648386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.648369 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:32:31.699321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.699293 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:31.701834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.701719 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:31.702980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:31.702965 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:32:32.019016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:32.018838 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:34.260415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:34.259373 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:34.565582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:34.565478 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:34.884563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:34.884473 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.201585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:35.201545 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.542835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:35.542728 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:35.602417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:35.566468 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.602417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:35.567208 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:32:35.889919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:35.889827 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:35.915417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:35.890790 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:32:36.232920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:36.206143 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.232920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:36.229385 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:32:36.307418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:36.264264 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.450429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:36.380194 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 29 22:32:36.613180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:36.613082 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:36.703435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:36.670715 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:36.824423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:36.823818 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:32:36.916501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:36.890813 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:32:37.230608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:37.230563 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:37.264553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:37.264499 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:32:37.265987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:37.265961 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:32:37.725336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:37.721723 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:32:37.725336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:37.721801 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:32:37.726347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:37.726321 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:32:38.193255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:38.193227 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:39.822926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:39.822893 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:41.302681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:41.302644 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:42.822025 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:42.821987 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:43.715774 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:32:43.715737 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:32:44.542730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:44.542686 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:46.220333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:46.220290 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:47.832968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:47.832939 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:48.491143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.491108 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9"] Oct 29 22:32:48.497126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.497101 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.531037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.530991 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9"] Oct 29 22:32:48.586366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586327 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.586366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586367 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.586611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586462 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vh85\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-kube-api-access-6vh85\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.586611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586518 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.586680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586645 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.586720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586693 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.586835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586815 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.586904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586880 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.586963 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.586947 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.688479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688443 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.688671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688497 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.688671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688544 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vh85\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-kube-api-access-6vh85\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.688671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688606 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.688671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688661 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.689029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688694 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.689029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688743 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.689029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688827 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.689029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.688863 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.689661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.689635 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istiod-ca-cert\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.690653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.690501 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-certs\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.690653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.690616 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-data\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.691858 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.691813 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-credential-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.692211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.692191 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-socket\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.692477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.692430 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-podinfo\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.696897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.696780 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-envoy\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.707298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.707270 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-token\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.708180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.708156 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vh85\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-kube-api-access-6vh85\") pod \"e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:48.815979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:48.815898 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:49.125580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.125501 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47"] Oct 29 22:32:49.131161 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.131138 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.152024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.151998 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9"] Oct 29 22:32:49.152148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.152044 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:49.152148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.152129 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:49.152262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.152178 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:49.171423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.165455 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47"] Oct 29 22:32:49.171423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.170093 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq"] Oct 29 22:32:49.180371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.176979 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.198197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.198160 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.198346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.198217 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.198346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.198279 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017d7fa-514b-426a-b489-a34942e99745-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.199173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199141 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.199273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199203 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.199273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199267 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.199407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199327 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.199465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199427 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.199465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199456 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.199575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199494 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqg5\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-kube-api-access-qrqg5\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.199575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199529 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.199827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199804 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.199979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.199957 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949978bd-024c-4b88-849f-e89bd7f191d8-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.200103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.200078 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.200217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.200152 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017d7fa-514b-426a-b489-a34942e99745-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.200217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.200197 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcpwn\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-kube-api-access-pcpwn\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.200297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.200227 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.200297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.200251 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949978bd-024c-4b88-849f-e89bd7f191d8-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.202625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.202602 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq"] Oct 29 22:32:49.303196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.303156 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcpwn\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-kube-api-access-pcpwn\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.303370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.303228 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.303370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.303258 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949978bd-024c-4b88-849f-e89bd7f191d8-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.303370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.303292 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.303370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.303326 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.303627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.303404 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017d7fa-514b-426a-b489-a34942e99745-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.304867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.304493 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949978bd-024c-4b88-849f-e89bd7f191d8-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.304867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.304661 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.304867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.304711 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.304867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.304777 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.304867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.304842 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.305211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.304935 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.305211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.304961 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.305315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.305291 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqg5\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-kube-api-access-qrqg5\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.305368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.305337 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.305485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.305467 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.305555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.305534 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949978bd-024c-4b88-849f-e89bd7f191d8-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.305617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.305575 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.305670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.305655 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017d7fa-514b-426a-b489-a34942e99745-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.305722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.305697 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.306087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.306061 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.306087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.306083 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.306307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.306284 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-certs\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.306568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.306506 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.306976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.305469 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.307117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.306960 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017d7fa-514b-426a-b489-a34942e99745-istiod-ca-cert\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.307212 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.307137 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-data\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.307636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.307198 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-credential-socket\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.309923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.309901 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949978bd-024c-4b88-849f-e89bd7f191d8-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.312895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.312870 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.314201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.314177 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-envoy\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.315805 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.315785 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017d7fa-514b-426a-b489-a34942e99745-istio-podinfo\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.331116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.331089 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.331252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.331228 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqg5\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-kube-api-access-qrqg5\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.331710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.331689 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-istio-token\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.332634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.332614 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcpwn\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-kube-api-access-pcpwn\") pod \"e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.446850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.446819 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:49.505197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.504931 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:49.584079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.583187 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:49.858503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.852133 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47"] Oct 29 22:32:49.858503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.854004 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" event={"ID":"27ff1219-04a6-4395-b0e1-2ddc2cfb534a","Type":"ContainerStarted","Data":"4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b"} Oct 29 22:32:49.858503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.854057 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" event={"ID":"27ff1219-04a6-4395-b0e1-2ddc2cfb534a","Type":"ContainerStarted","Data":"bf5c937af86234b8a3af3f05c003fe5a9db4cc7ea1a5eab6b81eb0b452088d99"} Oct 29 22:32:49.863025 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:32:49.862991 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949978bd_024c_4b88_849f_e89bd7f191d8.slice/crio-1488db3f7066eadc9fe3bc60353f6d36c9e9e808d7a1b9ac33d6e8086d244087 WatchSource:0}: Error finding container 1488db3f7066eadc9fe3bc60353f6d36c9e9e808d7a1b9ac33d6e8086d244087: Status 404 returned error can't find the container with id 1488db3f7066eadc9fe3bc60353f6d36c9e9e808d7a1b9ac33d6e8086d244087 Oct 29 22:32:49.866186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.866148 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:49.866269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.866231 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:49.866309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.866272 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:49.936442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:49.934045 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podStartSLOduration=1.934025036 podStartE2EDuration="1.934025036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:49.896694195 +0000 UTC m=+1032.378011689" watchObservedRunningTime="2026-10-29 22:32:49.934025036 +0000 UTC m=+1032.415342530" Oct 29 22:32:50.119082 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:32:50.119043 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3017d7fa_514b_426a_b489_a34942e99745.slice/crio-8e741f5ac1416e24ed5cc9400ecba758d2142e51dd0d180fe1dd1cdef3cf40c5 WatchSource:0}: Error finding container 8e741f5ac1416e24ed5cc9400ecba758d2142e51dd0d180fe1dd1cdef3cf40c5: Status 404 returned error can't find the container with id 8e741f5ac1416e24ed5cc9400ecba758d2142e51dd0d180fe1dd1cdef3cf40c5 Oct 29 22:32:50.121453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.121422 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq"] Oct 29 22:32:50.130532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.125777 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:50.130532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.125926 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:50.130532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.126026 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:32:50.826734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.826635 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:50.828365 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.828333 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:32:50.828501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.828424 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:32:50.890196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.888885 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" event={"ID":"3017d7fa-514b-426a-b489-a34942e99745","Type":"ContainerStarted","Data":"fdb70db627efec49f6fd61bc2c108f8c087dc3239fed979801b179a640efbf1e"} Oct 29 22:32:50.890196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.888964 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" event={"ID":"3017d7fa-514b-426a-b489-a34942e99745","Type":"ContainerStarted","Data":"8e741f5ac1416e24ed5cc9400ecba758d2142e51dd0d180fe1dd1cdef3cf40c5"} Oct 29 22:32:50.896560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.896462 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" event={"ID":"949978bd-024c-4b88-849f-e89bd7f191d8","Type":"ContainerStarted","Data":"f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3"} Oct 29 22:32:50.896560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.896525 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" event={"ID":"949978bd-024c-4b88-849f-e89bd7f191d8","Type":"ContainerStarted","Data":"1488db3f7066eadc9fe3bc60353f6d36c9e9e808d7a1b9ac33d6e8086d244087"} Oct 29 22:32:50.939299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.939243 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podStartSLOduration=1.939226686 podStartE2EDuration="1.939226686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:50.935980461 +0000 UTC m=+1033.417297955" watchObservedRunningTime="2026-10-29 22:32:50.939226686 +0000 UTC m=+1033.420544179" Oct 29 22:32:50.989138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:50.989071 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podStartSLOduration=1.98904817 podStartE2EDuration="1.98904817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:32:50.986920404 +0000 UTC m=+1033.468237897" watchObservedRunningTime="2026-10-29 22:32:50.98904817 +0000 UTC m=+1033.470365664" Oct 29 22:32:51.450649 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.450611 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:51.451307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.451285 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:51.451511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.451476 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:32:51.511188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.509691 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:51.512580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.512555 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:51.512769 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.512744 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 22:32:51.768008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.767914 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:51.818864 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.818829 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:32:51.819167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:51.819137 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:32:52.448145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:52.448110 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:52.448442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:52.448411 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:32:52.506582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:52.506552 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:52.529417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:52.522286 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 22:32:52.818047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:52.817946 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:32:52.818331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:52.818287 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:32:53.483417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:53.471508 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:53.483417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:53.471575 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:32:53.558124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:53.556443 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:53.558124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:53.556507 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 22:32:53.819797 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:53.819713 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:32:53.820236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:53.820204 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:32:54.404475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:54.404324 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:54.448010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:54.447970 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:54.449211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:54.448274 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:32:54.505811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:54.505774 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:54.506096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:54.506064 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 22:32:54.841677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:54.841101 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:32:54.841677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:54.841163 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:32:55.450280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:55.450243 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:55.450593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:55.450561 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:32:55.508832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:55.508797 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:55.509114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:55.509086 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 22:32:55.821021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:55.820921 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:32:55.821286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:55.821245 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:32:56.447716 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:56.447675 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:56.448174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:56.447795 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:32:56.515885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:56.515849 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:56.516161 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:56.516136 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 22:32:56.535715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:56.535682 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:56.833927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:56.833833 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 29 22:32:56.834192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:56.834162 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 29 22:32:57.453128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:57.453092 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 29 22:32:57.453768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:57.453734 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 29 22:32:57.512630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:57.512576 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 29 22:32:57.512948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:57.512904 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 29 22:32:58.063136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:58.063102 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:58.064458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:58.064428 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:58.065985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:58.065967 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:32:58.360540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:58.360439 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:32:59.000712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:59.000656 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:59.001436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:59.001372 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:59.002669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:59.002381 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:32:59.131325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:59.131149 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:59.136382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:59.136355 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:32:59.141682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:32:59.141657 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:33:00.000882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:00.000846 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:01.815838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:01.815799 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:04.661413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:04.658495 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:04.721820 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:04.721769 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:04.921628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:04.921518 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:05.437548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:05.437499 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:05.664998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:05.664956 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:05.742352 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:05.724561 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:05.851159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:05.851123 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:33:05.915426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:05.888285 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:33:05.940534 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:05.940482 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:05.941676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:05.941651 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:33:06.446753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:06.446694 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.448483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:06.448443 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.899486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:06.500684 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:33:06.899486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:06.830951 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:06.899486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:06.857668 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:06.949452 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:06.949405 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.512428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:07.512043 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.926628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:07.926517 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:07.931413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:07.927561 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:07.990609 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:07.990568 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:08.180039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:08.039333 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:08.187605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:08.187553 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.520203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:08.520156 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:08.688677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:08.677291 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:08.713506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:08.713464 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:08.713782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:08.713749 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:08.897478 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:33:08.871539 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.138s" Oct 29 22:33:08.995475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:08.937536 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.001427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:08.937625 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:09.205353 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:09.205311 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.340987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:09.340934 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:09.341292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:09.341250 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:09.535471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:09.534212 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:09.998670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:09.997147 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:10.113580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:10.113528 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:10.235418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:10.232178 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:10.536573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:10.536512 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.129418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.125490 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.222247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.222199 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.292459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.239240 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 29 22:33:11.292791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.243227 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.580446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.573578 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.605095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.602511 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:11.605095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.602577 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.607507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.607456 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:11.867622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.867524 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:11.867897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:11.867865 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.145265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:12.144023 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.226732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:12.226691 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.297259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:12.297213 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.664523 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:12.664473 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.723610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:12.686588 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:12.723610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:12.707903 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:12.723610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:12.722553 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:13.028422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.026471 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:13.028422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.026532 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:13.147595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.147543 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:13.230355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.230312 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:13.300136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.299291 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:13.431565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.431524 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:13.431884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.431835 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:13.593598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.593501 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:13.593866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.593836 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:13.718626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:13.718557 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:14.097588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:14.097545 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:14.097900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:14.097865 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:14.148563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:14.148514 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:14.224669 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:33:14.224627 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:33:14.246418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:14.237867 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:14.302418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:14.301945 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:14.741418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:14.739448 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:14.828419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:14.827518 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:14.828419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:14.827582 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:15.050666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.050506 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:15.051216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.050820 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:15.155539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.155488 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:15.263422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.243533 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:15.263422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.260552 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:15.263422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.260626 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:15.317635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.317531 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:15.476414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.467915 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:33:15.524663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.524634 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:33:15.533674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.533647 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:33:15.554202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:15.554172 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:33:16.236332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:16.236294 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:17.963689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:17.949349 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:18.698360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.698321 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm"] Oct 29 22:33:18.704971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.704939 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.742914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.742883 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm"] Oct 29 22:33:18.842264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842225 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.842501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842294 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.842501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842387 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.842655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842531 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.842655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842626 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.842778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842756 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpsjc\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-kube-api-access-dpsjc\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.842852 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842835 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.842915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842893 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.842971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.842954 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.944829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.944796 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsjc\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-kube-api-access-dpsjc\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.944872 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.944891 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.944917 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.944946 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.944998 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.945037 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.945097 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.945166 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.945897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.945865 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.946186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.946148 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.946383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.946354 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.946512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.946410 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.946718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.946691 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.947926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.947900 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.951586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.951534 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.964984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.964960 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpsjc\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-kube-api-access-dpsjc\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:18.966675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:18.966653 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:19.044465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.044433 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:19.378782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.378749 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm"] Oct 29 22:33:19.379455 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:33:19.379422 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed25461_b10f_43f6_a689_a39a08fc0ac8.slice/crio-de59e460b0970d1f1f2bfe220e1bec0c157ab55ac5d3b48e3b09ca4e0fb5e5cd WatchSource:0}: Error finding container de59e460b0970d1f1f2bfe220e1bec0c157ab55ac5d3b48e3b09ca4e0fb5e5cd: Status 404 returned error can't find the container with id de59e460b0970d1f1f2bfe220e1bec0c157ab55ac5d3b48e3b09ca4e0fb5e5cd Oct 29 22:33:19.382483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.382441 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:33:19.382561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.382538 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:33:19.382608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.382582 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:33:19.464368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.464339 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" event={"ID":"3ed25461-b10f-43f6-a689-a39a08fc0ac8","Type":"ContainerStarted","Data":"83308433f77d2127e2a70c9f45b55273c9e4f478da789130bd1676c108ea75d7"} Oct 29 22:33:19.464553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.464435 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" event={"ID":"3ed25461-b10f-43f6-a689-a39a08fc0ac8","Type":"ContainerStarted","Data":"de59e460b0970d1f1f2bfe220e1bec0c157ab55ac5d3b48e3b09ca4e0fb5e5cd"} Oct 29 22:33:19.511285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.511224 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podStartSLOduration=1.511206125 podStartE2EDuration="1.511206125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:19.508217407 +0000 UTC m=+1061.989534901" watchObservedRunningTime="2026-10-29 22:33:19.511206125 +0000 UTC m=+1061.992523619" Oct 29 22:33:19.604475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:19.602809 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:20.045290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:20.045255 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:20.047019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:20.046973 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:33:20.047172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:20.047050 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:33:21.045255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:21.045221 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:33:21.045454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:21.045275 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:33:21.356543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:21.356456 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:22.045001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:22.044966 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:33:22.045183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:22.045030 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:33:23.045422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:23.040985 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:23.064434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:23.047183 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:33:23.064434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:23.047253 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:33:24.045461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:24.045388 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:33:24.045959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:24.045461 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:33:24.735640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:24.735603 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:25.045499 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:25.045405 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:33:25.045869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:25.045516 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:33:26.045745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:26.045698 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 29 22:33:26.046262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:26.045777 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 29 22:33:26.444290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:26.444257 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:26.969201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:26.969150 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:27.122194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:27.122151 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:27.129235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:27.129206 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:27.130229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:27.130209 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:27.133572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:27.133553 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:33:27.996520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:27.996479 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:29.647745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:29.647704 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:31.425061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:31.424945 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:32.944880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:32.944838 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:34.606029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:34.605976 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:36.098184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:36.098145 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:36.704307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:36.704274 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:37.871089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:37.871042 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:39.325421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:39.314304 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:39.640478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:39.634534 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.923883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:39.923777 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:39.947576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:39.947532 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:40.315601 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:40.315512 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:40.317677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:40.316465 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:33:40.655506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:40.647474 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:40.655506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:40.648190 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:33:40.984419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:40.966061 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.014421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:40.988705 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:33:41.014421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:40.993303 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:41.014421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:40.994973 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:33:41.189419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:41.188511 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 29 22:33:41.335598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:41.335492 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:41.591539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:41.589273 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 29 22:33:41.669105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:41.669063 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:41.969538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:41.969497 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.006836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.006792 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.202546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.194834 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:42.202546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.194898 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.440577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.405870 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.694478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.690499 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:42.694478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.690594 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.694478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.691475 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:42.762556 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.750919 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:42.999488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:42.997741 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:43.017504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:43.008263 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.407236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:43.407114 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:43.533012 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:43.532970 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.533302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:43.533267 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.533948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:43.533921 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:43.537482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:43.537449 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:43.817421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:43.792647 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.017540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.017493 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.018237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.017796 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.123596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.123468 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:44.127561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.127511 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.244537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.244490 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:44.244729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.244561 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.412185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.412085 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.837420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.835492 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.966680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.900626 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:44.966680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.900687 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:44.997064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:44.997028 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:45.022925 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.022880 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.023257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.023230 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:45.065526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.025455 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:33:45.107485 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:33:45.081127 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:33:45.107846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.083479 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:45.135214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.135171 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.399474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.380221 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:45.399788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.399756 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:45.460449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.433325 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:45.852432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:45.837696 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:46.026682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:46.026624 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:46.173703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:46.173598 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:46.392457 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:46.388126 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 29 22:33:46.460922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:46.459983 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:47.021417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:47.001504 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:47.034372 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:47.034323 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:33:47.095995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:47.095649 5610 patch_prober.go:29] interesting pod/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:33:47.095995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:47.095715 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:47.479084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:47.479034 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:33:47.740965 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:47.740852 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:48.138491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:48.100676 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:33:48.334840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:48.334799 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:48.375091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:48.374944 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:33:48.442961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:48.442466 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:33:48.442961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:48.442526 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:33:48.442961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:48.442559 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:33:48.445445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:48.445163 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:33:49.835925 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:49.835572 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:51.340219 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:51.340160 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:52.996616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:52.996571 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:54.569171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:54.569132 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:55.079983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:55.079943 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:55.240242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:55.240203 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:55.739146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:55.739097 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:55.982387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:55.982347 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:56.210986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.210931 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:56.513949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.513861 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv"] Oct 29 22:33:56.520200 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.520170 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.539261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.539231 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1309d608-openshift-default-dockercfg-7dpb2\"" Oct 29 22:33:56.566749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.566715 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv"] Oct 29 22:33:56.670693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.670651 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:56.884361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.884065 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.886377 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-socket\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.886554 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-credential-socket\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.886708 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-data\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.886771 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-certs\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.886842 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xsgz\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-kube-api-access-8xsgz\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.886887 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-token\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.887094 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9cf790-8608-4f83-b20d-fc09ef1011f7-istiod-ca-cert\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.887163 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-podinfo\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:56.893604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:56.887196 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-envoy\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.007544 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.007505 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-socket\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.009148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.009120 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-credential-socket\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.009921 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.009809 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-data\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.010421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.009629 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-credential-socket\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.010571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.008603 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-socket\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.010667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.010302 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-data\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.010771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.010366 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-certs\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.029258 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.029222 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xsgz\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-kube-api-access-8xsgz\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.029932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.029893 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-token\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.030604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.030582 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9cf790-8608-4f83-b20d-fc09ef1011f7-istiod-ca-cert\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.031496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.031477 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-podinfo\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.032076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.032056 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-envoy\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.032753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.011118 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-certs\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.032906 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.031375 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9cf790-8608-4f83-b20d-fc09ef1011f7-istiod-ca-cert\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.038057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.038034 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-envoy\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.071500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.041243 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-podinfo\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.071500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.064301 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xsgz\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-kube-api-access-8xsgz\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.071500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.066188 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-token\") pod \"e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.156470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.150768 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:57.583532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.583492 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv"] Oct 29 22:33:57.588348 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:33:57.588323 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9cf790_8608_4f83_b20d_fc09ef1011f7.slice/crio-b95affc1f3611f2719d98beea0966886fe6ba04ddc18e178ac6436f36ff9f3bd WatchSource:0}: Error finding container b95affc1f3611f2719d98beea0966886fe6ba04ddc18e178ac6436f36ff9f3bd: Status 404 returned error can't find the container with id b95affc1f3611f2719d98beea0966886fe6ba04ddc18e178ac6436f36ff9f3bd Oct 29 22:33:57.591409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.591355 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:33:57.591515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.591469 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:33:57.591580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:57.591519 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:33:58.276791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:58.276754 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:58.512667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:58.512633 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" event={"ID":"af9cf790-8608-4f83-b20d-fc09ef1011f7","Type":"ContainerStarted","Data":"fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f"} Oct 29 22:33:58.512850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:58.512688 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" event={"ID":"af9cf790-8608-4f83-b20d-fc09ef1011f7","Type":"ContainerStarted","Data":"b95affc1f3611f2719d98beea0966886fe6ba04ddc18e178ac6436f36ff9f3bd"} Oct 29 22:33:59.157813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:59.157773 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:59.164227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:59.164199 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:59.214187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:59.214123 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" podStartSLOduration=3.214103035 podStartE2EDuration="3.214103035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:33:58.577354648 +0000 UTC m=+1101.058672141" watchObservedRunningTime="2026-10-29 22:33:59.214103035 +0000 UTC m=+1101.695420529" Oct 29 22:33:59.520274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:59.520239 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:59.521617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:59.521593 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:33:59.617579 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:59.616962 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:33:59.789450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:33:59.789311 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:00.188463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:00.185989 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:00.358477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:00.358431 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:00.539437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:00.539373 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:01.944016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:01.943972 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.076842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.076806 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c"] Oct 29 22:34:02.081910 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.081881 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:02.104930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.104889 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.108218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.108191 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1309d608\"" Oct 29 22:34:02.114135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.114110 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c"] Oct 29 22:34:02.119582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.119561 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1309d608-dockercfg-dhlvr\"" Oct 29 22:34:02.265545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.249454 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzsd\" (UniqueName: \"kubernetes.io/projected/3c7dc27d-60b6-44e7-a7a0-f731807890a1-kube-api-access-dtzsd\") pod \"payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c\" (UID: \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:02.345427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.342677 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7dc27d-60b6-44e7-a7a0-f731807890a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c\" (UID: \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:02.345427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.308466 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph"] Oct 29 22:34:02.427469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.419119 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:02.446569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.446541 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:02.446815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.446781 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7dc27d-60b6-44e7-a7a0-f731807890a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c\" (UID: \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:02.447029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.447011 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzsd\" (UniqueName: \"kubernetes.io/projected/3c7dc27d-60b6-44e7-a7a0-f731807890a1-kube-api-access-dtzsd\") pod \"payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c\" (UID: \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:02.454512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.448245 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7dc27d-60b6-44e7-a7a0-f731807890a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c\" (UID: \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:02.498473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.492156 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzsd\" (UniqueName: \"kubernetes.io/projected/3c7dc27d-60b6-44e7-a7a0-f731807890a1-kube-api-access-dtzsd\") pod \"payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c\" (UID: \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:02.498473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.492223 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph"] Oct 29 22:34:02.588472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.584358 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgd88\" (UniqueName: \"kubernetes.io/projected/f937a2f3-176a-4d9d-8615-96a325e62ebb-kube-api-access-dgd88\") pod \"payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph\" (UID: \"f937a2f3-176a-4d9d-8615-96a325e62ebb\") " pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:02.598417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.588994 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937a2f3-176a-4d9d-8615-96a325e62ebb-plugins-config-volume\") pod \"payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph\" (UID: \"f937a2f3-176a-4d9d-8615-96a325e62ebb\") " pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:02.690778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.690740 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgd88\" (UniqueName: \"kubernetes.io/projected/f937a2f3-176a-4d9d-8615-96a325e62ebb-kube-api-access-dgd88\") pod \"payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph\" (UID: \"f937a2f3-176a-4d9d-8615-96a325e62ebb\") " pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:02.690986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.690922 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937a2f3-176a-4d9d-8615-96a325e62ebb-plugins-config-volume\") pod \"payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph\" (UID: \"f937a2f3-176a-4d9d-8615-96a325e62ebb\") " pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:02.691726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.691700 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937a2f3-176a-4d9d-8615-96a325e62ebb-plugins-config-volume\") pod \"payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph\" (UID: \"f937a2f3-176a-4d9d-8615-96a325e62ebb\") " pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:02.706181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.706158 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:02.710633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.710605 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgd88\" (UniqueName: \"kubernetes.io/projected/f937a2f3-176a-4d9d-8615-96a325e62ebb-kube-api-access-dgd88\") pod \"payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph\" (UID: \"f937a2f3-176a-4d9d-8615-96a325e62ebb\") " pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:02.978068 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:02.978034 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:03.014771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.014739 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c"] Oct 29 22:34:03.017381 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:34:03.017349 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7dc27d_60b6_44e7_a7a0_f731807890a1.slice/crio-86cb5129c42d92acc6458f6f1b1f6040d6344ba113861fce3284a24f76255846 WatchSource:0}: Error finding container 86cb5129c42d92acc6458f6f1b1f6040d6344ba113861fce3284a24f76255846: Status 404 returned error can't find the container with id 86cb5129c42d92acc6458f6f1b1f6040d6344ba113861fce3284a24f76255846 Oct 29 22:34:03.054662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.054366 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:03.194130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.194090 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:03.293020 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:34:03.292994 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf937a2f3_176a_4d9d_8615_96a325e62ebb.slice/crio-9e7c3749709f9b53773fe4f8fddb4189ed2cbf50c5df93d1b00ea88f1d0cb1dc WatchSource:0}: Error finding container 9e7c3749709f9b53773fe4f8fddb4189ed2cbf50c5df93d1b00ea88f1d0cb1dc: Status 404 returned error can't find the container with id 9e7c3749709f9b53773fe4f8fddb4189ed2cbf50c5df93d1b00ea88f1d0cb1dc Oct 29 22:34:03.294374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.294299 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph"] Oct 29 22:34:03.379016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.378981 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:03.532264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.532231 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:03.637359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.637285 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" event={"ID":"3c7dc27d-60b6-44e7-a7a0-f731807890a1","Type":"ContainerStarted","Data":"f3790d088fcc46a71826c8e99fe018055fe15ff828ce6465216c82f9998ae750"} Oct 29 22:34:03.637359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.637338 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:03.637359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.637357 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:03.637596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.637373 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" event={"ID":"3c7dc27d-60b6-44e7-a7a0-f731807890a1","Type":"ContainerStarted","Data":"86cb5129c42d92acc6458f6f1b1f6040d6344ba113861fce3284a24f76255846"} Oct 29 22:34:03.637596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.637387 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" event={"ID":"f937a2f3-176a-4d9d-8615-96a325e62ebb","Type":"ContainerStarted","Data":"6aac864412620c2464abe86b00ebad63437c7e42c855f8a4b61e37d90101da50"} Oct 29 22:34:03.637596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.637422 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" event={"ID":"f937a2f3-176a-4d9d-8615-96a325e62ebb","Type":"ContainerStarted","Data":"9e7c3749709f9b53773fe4f8fddb4189ed2cbf50c5df93d1b00ea88f1d0cb1dc"} Oct 29 22:34:03.679356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.679294 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" podStartSLOduration=1.679274791 podStartE2EDuration="1.679274791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:03.676840082 +0000 UTC m=+1106.158157601" watchObservedRunningTime="2026-10-29 22:34:03.679274791 +0000 UTC m=+1106.160592285" Oct 29 22:34:03.711995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.711963 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:03.722609 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.722553 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" podStartSLOduration=1.722538722 podStartE2EDuration="1.722538722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:03.718248051 +0000 UTC m=+1106.199565546" watchObservedRunningTime="2026-10-29 22:34:03.722538722 +0000 UTC m=+1106.203856214" Oct 29 22:34:03.882730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:03.880537 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:04.595407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.595357 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n"] Oct 29 22:34:04.600518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.600494 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.616839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.616803 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3593b8ba-gw-openshift-default-dockercfg-skwqf\"" Oct 29 22:34:04.641310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.640554 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n"] Oct 29 22:34:04.690209 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.690168 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.690415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.690261 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkh4\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-kube-api-access-2fkh4\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.690532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.690511 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.690731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.690700 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.690807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.690775 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.690870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.690853 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.690945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.690929 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.690981 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.690962 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.691097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.691081 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.699597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.699527 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:04.795683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.795636 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.821014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.820982 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkh4\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-kube-api-access-2fkh4\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.821686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.821659 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.821815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.821770 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.821870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.821824 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.821942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.821883 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.821999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.821949 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.821999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.821982 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.822095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.822060 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.822930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.822891 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.823158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.823135 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-credential-socket\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.823264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.823156 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-certs\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.823640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.823618 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-data\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.824825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.824584 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istiod-ca-cert\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.830896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.830867 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-podinfo\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.836704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.836674 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-envoy\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.850475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.844105 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkh4\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-kube-api-access-2fkh4\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.857130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.857093 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-token\") pod \"e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:04.922900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:04.922856 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:05.217740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.217634 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:05.337941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.337911 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n"] Oct 29 22:34:05.339775 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.339732 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:34:05.339880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.339818 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:34:05.339880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.339855 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:34:05.661282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.661190 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" event={"ID":"43bce1d9-bceb-4c3a-9cc5-67de580eace1","Type":"ContainerStarted","Data":"67a51e8f740f07ebcc4ecd69c8d30880e12423407be72f46d8d187d6f563e399"} Oct 29 22:34:05.661282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.661247 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" event={"ID":"43bce1d9-bceb-4c3a-9cc5-67de580eace1","Type":"ContainerStarted","Data":"59be70b8b347f94473b7fd1c493113b028f7bcf69b5e952c2edea1ace2f9aad4"} Oct 29 22:34:05.708419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.702947 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:05.748477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.748192 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" podStartSLOduration=1.748167775 podStartE2EDuration="1.748167775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:05.730894022 +0000 UTC m=+1108.212211515" watchObservedRunningTime="2026-10-29 22:34:05.748167775 +0000 UTC m=+1108.229485269" Oct 29 22:34:05.939474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:05.933483 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:06.931127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:06.931091 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:07.729418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:07.721413 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:07.823415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:07.813614 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:07.823415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:07.822141 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:34:08.542417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:08.523625 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:09.432419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:09.431783 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:09.734428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:09.724528 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.012588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.011548 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.052271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.030023 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:10.225421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.222243 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:10.278511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.238013 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:10.352442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.330868 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv"] Oct 29 22:34:10.462419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.433666 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.524926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.513946 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv"] Oct 29 22:34:10.525250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.525222 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:34:10.564480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.558576 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg"] Oct 29 22:34:10.591663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.514047 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:10.682444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.641904 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65a0c50-138d-48dc-94f8-ff440f010f56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv\" (UID: \"d65a0c50-138d-48dc-94f8-ff440f010f56\") " pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:10.682787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.682747 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hdz\" (UniqueName: \"kubernetes.io/projected/d65a0c50-138d-48dc-94f8-ff440f010f56-kube-api-access-z4hdz\") pod \"payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv\" (UID: \"d65a0c50-138d-48dc-94f8-ff440f010f56\") " pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:10.689581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.687766 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:10.700752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.700722 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3593b8ba\"" Oct 29 22:34:10.713851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.713822 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3593b8ba-dockercfg-4kw9d\"" Oct 29 22:34:10.727732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.727648 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg"] Oct 29 22:34:10.794418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.766787 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:10.820880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.820043 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hdz\" (UniqueName: \"kubernetes.io/projected/d65a0c50-138d-48dc-94f8-ff440f010f56-kube-api-access-z4hdz\") pod \"payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv\" (UID: \"d65a0c50-138d-48dc-94f8-ff440f010f56\") " pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:10.821038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.820896 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqj6\" (UniqueName: \"kubernetes.io/projected/bb22f6e9-4827-4e18-942a-458f7073f63c-kube-api-access-sxqj6\") pod \"payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg\" (UID: \"bb22f6e9-4827-4e18-942a-458f7073f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:10.821098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.821083 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb22f6e9-4827-4e18-942a-458f7073f63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg\" (UID: \"bb22f6e9-4827-4e18-942a-458f7073f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:10.825413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.821374 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:34:10.840429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.821896 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65a0c50-138d-48dc-94f8-ff440f010f56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv\" (UID: \"d65a0c50-138d-48dc-94f8-ff440f010f56\") " pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:10.877029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.845525 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65a0c50-138d-48dc-94f8-ff440f010f56-plugins-config-volume\") pod \"payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv\" (UID: \"d65a0c50-138d-48dc-94f8-ff440f010f56\") " pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:10.973425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:10.972464 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hdz\" (UniqueName: \"kubernetes.io/projected/d65a0c50-138d-48dc-94f8-ff440f010f56-kube-api-access-z4hdz\") pod \"payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv\" (UID: \"d65a0c50-138d-48dc-94f8-ff440f010f56\") " pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:11.052420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.051092 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:11.074470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.070807 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:34:11.192713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.156888 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 29 22:34:11.265591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.228757 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.265591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.229468 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:34:11.454847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.454040 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqj6\" (UniqueName: \"kubernetes.io/projected/bb22f6e9-4827-4e18-942a-458f7073f63c-kube-api-access-sxqj6\") pod \"payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg\" (UID: \"bb22f6e9-4827-4e18-942a-458f7073f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:11.454847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.454189 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb22f6e9-4827-4e18-942a-458f7073f63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg\" (UID: \"bb22f6e9-4827-4e18-942a-458f7073f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:11.477519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.457613 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb22f6e9-4827-4e18-942a-458f7073f63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg\" (UID: \"bb22f6e9-4827-4e18-942a-458f7073f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:11.477519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.457792 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:11.533350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.533310 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:11.693417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.668224 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.693417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.682182 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:11.693417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.682229 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:11.847653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:11.847545 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:12.440425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.368543 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:12.609639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.581626 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:12.609639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.581705 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.609639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.582475 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.726447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.706045 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.811365 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.811306 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.882272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.843906 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:12.903665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.903620 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:12.929607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:12.883528 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.202123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.202090 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqj6\" (UniqueName: \"kubernetes.io/projected/bb22f6e9-4827-4e18-942a-458f7073f63c-kube-api-access-sxqj6\") pod \"payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg\" (UID: \"bb22f6e9-4827-4e18-942a-458f7073f63c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:13.378445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.373483 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:13.416425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.403746 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:13.477477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.420492 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:13.477477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.459656 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.610185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.609660 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:13.669532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.669490 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:13.669843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.669811 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:13.853474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.852931 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:13.989420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:13.981429 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:14.122083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.121982 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:14.122383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.122335 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:14.386479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.377119 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:14.469427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.468677 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:14.508473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.468775 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:14.888596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.888538 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:14.963541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.963505 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:14.963859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.963808 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:14.964032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.963680 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:14.964170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:14.964150 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:15.101417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.097557 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:15.214615 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.214576 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv"] Oct 29 22:34:15.243528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.243490 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:15.287533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.268576 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:34:15.287533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.270043 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:34:15.338266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.338218 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:15.375441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.338545 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:15.409475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.409430 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:15.624064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.624038 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:34:15.649804 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:34:15.649765 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:34:15.957578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:15.950742 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:16.144809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:16.144701 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:16.352205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:16.352161 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:16.423141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:16.423045 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:34:16.423460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:16.423375 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:34:16.426078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:16.426011 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:34:16.444147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:16.444035 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" event={"ID":"d65a0c50-138d-48dc-94f8-ff440f010f56","Type":"ContainerStarted","Data":"be66d28535e0a133c4da6eecc7e9d411070db92282fbb6cb65b731dda118269a"} Oct 29 22:34:16.450986 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:34:16.450960 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb22f6e9_4827_4e18_942a_458f7073f63c.slice/crio-a0464d94d0071e7fe738390c0937ecd2c7d2c73483090225716616cf0ddcf9e5 WatchSource:0}: Error finding container a0464d94d0071e7fe738390c0937ecd2c7d2c73483090225716616cf0ddcf9e5: Status 404 returned error can't find the container with id a0464d94d0071e7fe738390c0937ecd2c7d2c73483090225716616cf0ddcf9e5 Oct 29 22:34:16.474650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:16.474099 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg"] Oct 29 22:34:16.679113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:16.679021 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:17.455278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.455236 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" event={"ID":"d65a0c50-138d-48dc-94f8-ff440f010f56","Type":"ContainerStarted","Data":"27fd3fecb6d81949507430ed2482251d17b7dc109540f20468f86a6c5497b402"} Oct 29 22:34:17.455920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.455355 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:17.457619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.457591 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" event={"ID":"bb22f6e9-4827-4e18-942a-458f7073f63c","Type":"ContainerStarted","Data":"a84f21561e82b8d93a06cb64064e2fa97a42ac39025dd7c99003c1d7e1d6e788"} Oct 29 22:34:17.457749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.457636 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" event={"ID":"bb22f6e9-4827-4e18-942a-458f7073f63c","Type":"ContainerStarted","Data":"a0464d94d0071e7fe738390c0937ecd2c7d2c73483090225716616cf0ddcf9e5"} Oct 29 22:34:17.457749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.457659 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:17.493796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.493747 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" podStartSLOduration=8.493734065 podStartE2EDuration="8.493734065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:17.492545693 +0000 UTC m=+1119.973863200" watchObservedRunningTime="2026-10-29 22:34:17.493734065 +0000 UTC m=+1119.975051558" Oct 29 22:34:17.531888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.531827 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" podStartSLOduration=8.531809303 podStartE2EDuration="8.531809303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:17.528111571 +0000 UTC m=+1120.009429065" watchObservedRunningTime="2026-10-29 22:34:17.531809303 +0000 UTC m=+1120.013126798" Oct 29 22:34:17.726030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.725934 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:17.984351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:17.982543 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:18.165418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:18.165131 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:18.505517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:18.505479 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:18.690324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:18.687617 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:18.971418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:18.967767 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:19.120470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:19.119962 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:19.264948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:19.264857 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:21.312441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:21.294097 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:21.992878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:21.992840 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:22.142777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:22.142741 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:22.997387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:22.997354 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:23.170264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:23.170231 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:23.332654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:23.332562 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:24.002503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:24.002436 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:25.712988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:25.712955 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:27.024843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:27.024806 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:27.163248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:27.163206 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:27.539320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:27.539273 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:27.740416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:27.723963 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:27.891563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:27.890974 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:28.472564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:28.472531 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:34:28.473067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:28.472980 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:34:28.945536 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:28.945496 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:30.415899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:30.415859 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:30.556637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:30.556602 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:30.722539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:30.714845 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:31.691436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:31.686000 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:32.556419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:32.537472 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:32.782418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:32.781067 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:33.409659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:33.409619 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:33.827417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:33.814658 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:33.827417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:33.815067 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:33.827417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:33.817211 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:33.876343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:33.861563 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.815419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:34.815257 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.815640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:34.815450 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.819416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:34.815810 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:34:34.819416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:34.815884 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:34:34.828417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:34.824705 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:34.828417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:34.825235 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:34:34.881715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:34.881673 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:34.886418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:34.883146 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:34:35.891450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:35.872436 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:35.891450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:35.882505 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:36.400554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:36.400503 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" podUID="f7c7e838-5a9d-477c-994e-e02cff87ba8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.487467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:36.426183 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:36.549328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:36.543106 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" podUID="f7c7e838-5a9d-477c-994e-e02cff87ba8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.565699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:36.565562 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 29 22:34:36.877431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:36.874506 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:36.910530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:36.910188 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.071420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:37.049987 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:37.454418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:37.423063 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:37.697418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:37.696376 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:37.877077 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:34:37.876980 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.033s" Oct 29 22:34:38.042419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:37.932535 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.073492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:38.062464 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:38.077840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:38.077786 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.158481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:38.133342 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:38.189420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:38.186535 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:38.189420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:38.186629 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:38.548438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:38.545633 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:38.703231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:38.703175 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:38.741373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:38.741338 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:39.086753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:39.086709 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:39.087948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:39.087912 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:39.117586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:39.110193 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:39.210196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:39.210157 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:39.707019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:39.706937 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:40.089231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:40.089106 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:40.115312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:40.114013 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:34:40.142476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:40.142432 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:40.160345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:40.160295 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:34:40.160789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:40.160631 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:40.661705 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:34:40.661666 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.024s" Oct 29 22:34:40.723611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:40.723562 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:41.099547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:41.099504 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:41.119611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:41.119553 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:41.205740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:41.149726 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:41.205740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:41.185351 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:34:41.581055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:41.580772 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:41.762112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:41.758096 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:34:42.136419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:42.135580 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:42.136419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:42.136007 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:42.196580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:42.196529 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:42.663426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:42.662495 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:42.763416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:42.759532 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:34:43.024738 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:43.024697 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:34:43.054114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:43.054083 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:34:43.394187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:43.369409 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:34:43.568695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:43.568144 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:34:43.569261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:43.569060 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:34:43.860739 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:43.858667 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:44.093730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:44.093690 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:44.523734 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:34:44.523689 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:34:45.679373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:45.679338 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:47.279869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:47.279816 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:48.642585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.642552 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk"] Oct 29 22:34:48.648695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.648671 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.668045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.668013 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk"] Oct 29 22:34:48.675722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.675685 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f922c6e9-serving-cert\"" Oct 29 22:34:48.705147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.705107 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-openshift-service-ca\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.705508 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.705472 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpq64\" (UniqueName: \"kubernetes.io/projected/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-kube-api-access-bpq64\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.705632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.705568 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-metrics-certs\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.705703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.705629 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.765354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.765318 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:48.807234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.807197 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-openshift-service-ca\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.807471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.807458 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpq64\" (UniqueName: \"kubernetes.io/projected/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-kube-api-access-bpq64\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.807552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.807516 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-metrics-certs\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.807610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.807573 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.807854 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:34:48.807837 5610 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-serving-cert: secret "maas-api-e2e-mt-f922c6e9-serving-cert" not found Oct 29 22:34:48.807957 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:34:48.807911 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls podName:5cb9529e-6f36-46e2-ac66-3168ef39d7bd nodeName:}" failed. No retries permitted until 2026-10-29 22:34:49.307891873 +0000 UTC m=+1151.789209360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls") pod "maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" (UID: "5cb9529e-6f36-46e2-ac66-3168ef39d7bd") : secret "maas-api-e2e-mt-f922c6e9-serving-cert" not found Oct 29 22:34:48.808099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.808063 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-openshift-service-ca\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.811378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.811345 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-metrics-certs\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:48.918024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.909778 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:48.923463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:48.922563 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpq64\" (UniqueName: \"kubernetes.io/projected/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-kube-api-access-bpq64\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:49.056786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:49.056671 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:49.332862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:49.332826 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:49.335260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:49.335234 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls\") pod \"maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:49.564020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:49.563986 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:49.889424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:49.889374 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk"] Oct 29 22:34:49.890560 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:34:49.890529 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb9529e_6f36_46e2_ac66_3168ef39d7bd.slice/crio-0795956eaf74bad12f9ca20fb9fade77ca1e32855bede31ccc6731482ab1ec6f WatchSource:0}: Error finding container 0795956eaf74bad12f9ca20fb9fade77ca1e32855bede31ccc6731482ab1ec6f: Status 404 returned error can't find the container with id 0795956eaf74bad12f9ca20fb9fade77ca1e32855bede31ccc6731482ab1ec6f Oct 29 22:34:49.892517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:49.892494 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:34:50.871716 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:50.871678 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" event={"ID":"5cb9529e-6f36-46e2-ac66-3168ef39d7bd","Type":"ContainerStarted","Data":"9966deeb3f4617d72900d64e7bb06bda0bf54a2db55e8a67cc1a0f6c2a6644f6"} Oct 29 22:34:50.871917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:50.871742 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" event={"ID":"5cb9529e-6f36-46e2-ac66-3168ef39d7bd","Type":"ContainerStarted","Data":"0795956eaf74bad12f9ca20fb9fade77ca1e32855bede31ccc6731482ab1ec6f"} Oct 29 22:34:50.871917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:50.871899 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:50.915448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:50.915383 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" podStartSLOduration=2.451228708 podStartE2EDuration="2.915368629s" totalImagesPullingTime="464.139921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:34:50.910455998 +0000 UTC m=+1153.391773491" watchObservedRunningTime="2026-10-29 22:34:50.915368629 +0000 UTC m=+1153.396686121" Oct 29 22:34:51.063419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:51.055939 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:52.521714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:52.521678 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:54.317116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:54.317080 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:55.979665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:55.979628 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:56.120414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:56.120188 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:56.886134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:56.886104 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:34:57.773418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:57.758033 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:59.361756 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.361721 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98"] Oct 29 22:34:59.366479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.366459 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.402542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.402520 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-591986bb-serving-cert\"" Oct 29 22:34:59.420920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.420890 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98"] Oct 29 22:34:59.525254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.525225 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-openshift-service-ca\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.525444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.525261 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-metrics-certs\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.525531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.525508 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.525575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.525565 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czl89\" (UniqueName: \"kubernetes.io/projected/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-kube-api-access-czl89\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.620816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.620718 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:59.627371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.627340 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.627527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.627431 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czl89\" (UniqueName: \"kubernetes.io/projected/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-kube-api-access-czl89\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.627527 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:34:59.627487 5610 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-serving-cert: secret "maas-api-e2e-derive-591986bb-serving-cert" not found Oct 29 22:34:59.627628 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:34:59.627547 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls podName:b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa nodeName:}" failed. No retries permitted until 2026-10-29 22:35:00.127531052 +0000 UTC m=+1162.608848522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls") pod "maas-api-e2e-derive-591986bb-9d44555f7-rnw98" (UID: "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa") : secret "maas-api-e2e-derive-591986bb-serving-cert" not found Oct 29 22:34:59.627741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.627723 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-openshift-service-ca\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.627809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.627751 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-metrics-certs\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.628434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.628410 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-openshift-service-ca\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.630215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.630194 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-metrics-certs\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.685417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.684858 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czl89\" (UniqueName: \"kubernetes.io/projected/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-kube-api-access-czl89\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:34:59.759138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.759099 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:34:59.900968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:34:59.900881 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:00.148974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:00.148940 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:35:00.151411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:00.151334 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls\") pod \"maas-api-e2e-derive-591986bb-9d44555f7-rnw98\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:35:00.281728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:00.281685 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:35:00.656438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:00.656381 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98"] Oct 29 22:35:00.657520 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:35:00.657495 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b6e8e8_a3b5_4fab_8a75_597dedc3dafa.slice/crio-1e905f7454d631fbe282518ed7f1b5bb48ee1661484c0a67af90e2c76cdab18f WatchSource:0}: Error finding container 1e905f7454d631fbe282518ed7f1b5bb48ee1661484c0a67af90e2c76cdab18f: Status 404 returned error can't find the container with id 1e905f7454d631fbe282518ed7f1b5bb48ee1661484c0a67af90e2c76cdab18f Oct 29 22:35:01.043447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:01.039290 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" event={"ID":"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa","Type":"ContainerStarted","Data":"1e905f7454d631fbe282518ed7f1b5bb48ee1661484c0a67af90e2c76cdab18f"} Oct 29 22:35:01.656159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:01.656080 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:02.050506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:02.050460 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" event={"ID":"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa","Type":"ContainerStarted","Data":"35210326513fa37d4bda9f0f201349536153071676176d8b885180c47d127e5b"} Oct 29 22:35:02.050912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:02.050526 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:35:02.094892 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:02.094831 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" podStartSLOduration=2.691675042 podStartE2EDuration="3.09481117s" totalImagesPullingTime="403.136128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:02.088795445 +0000 UTC m=+1164.570112962" watchObservedRunningTime="2026-10-29 22:35:02.09481117 +0000 UTC m=+1164.576128657" Oct 29 22:35:02.649421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:02.645463 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:03.543924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:03.543890 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:05.258647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:05.248221 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:07.027580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:07.027543 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:07.428361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:07.428280 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph"] Oct 29 22:35:07.458758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:07.458729 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c"] Oct 29 22:35:07.600304 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:07.600266 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:07.759422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:07.750930 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:08.115476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:08.105573 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:35:08.928524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:08.928486 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:12.131418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:12.113589 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.249416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:12.249300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:12.444479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:12.435508 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" podUID="f937a2f3-176a-4d9d-8615-96a325e62ebb" containerName="payload-processing" containerID="cri-o://6aac864412620c2464abe86b00ebad63437c7e42c855f8a4b61e37d90101da50" gracePeriod=25 Oct 29 22:35:12.470421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:12.466481 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" podUID="3c7dc27d-60b6-44e7-a7a0-f731807890a1" containerName="payload-pre-processing" containerID="cri-o://f3790d088fcc46a71826c8e99fe018055fe15ff828ce6465216c82f9998ae750" gracePeriod=25 Oct 29 22:35:12.662108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:12.661435 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.885620 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:12.885514 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:12.919533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:12.918491 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.067477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.063972 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr"] Oct 29 22:35:13.076279 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.076240 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c5630c-g2mvr"] Oct 29 22:35:13.128430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.117904 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:35:13.128430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.118340 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:35:13.551416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.550522 5610 patch_prober.go:29] interesting pod/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:35:13.551416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.550587 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.735418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.733136 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:13.889462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.888639 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:35:13.968466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:13.952338 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:35:14.227654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.202513 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:14.399351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.394421 5610 generic.go:356] "Generic (PLEG): container finished" podID="f937a2f3-176a-4d9d-8615-96a325e62ebb" containerID="6aac864412620c2464abe86b00ebad63437c7e42c855f8a4b61e37d90101da50" exitCode=0 Oct 29 22:35:14.679558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.646216 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb70b1f-546f-407c-a7f5-6dc0629c2e0b" path="/var/lib/kubelet/pods/bfb70b1f-546f-407c-a7f5-6dc0629c2e0b/volumes" Oct 29 22:35:14.717247 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:35:14.717212 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.084s" Oct 29 22:35:14.717569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.717540 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" event={"ID":"f937a2f3-176a-4d9d-8615-96a325e62ebb","Type":"ContainerDied","Data":"6aac864412620c2464abe86b00ebad63437c7e42c855f8a4b61e37d90101da50"} Oct 29 22:35:14.732421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.717759 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:35:14.733078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.733043 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" event={"ID":"3c7dc27d-60b6-44e7-a7a0-f731807890a1","Type":"ContainerDied","Data":"f3790d088fcc46a71826c8e99fe018055fe15ff828ce6465216c82f9998ae750"} Oct 29 22:35:14.806414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.805425 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:35:14.825666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.735690 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:35:14.934853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.934820 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:35:14.960435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.680730 5610 generic.go:356] "Generic (PLEG): container finished" podID="3c7dc27d-60b6-44e7-a7a0-f731807890a1" containerID="f3790d088fcc46a71826c8e99fe018055fe15ff828ce6465216c82f9998ae750" exitCode=0 Oct 29 22:35:14.960435 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:35:14.862435 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:35:14.960435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.937102 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:14.985424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.979965 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.013146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:14.999751 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:35:15.206836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.194655 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7dc27d-60b6-44e7-a7a0-f731807890a1-kube-api-access-dtzsd\" (UniqueName: \"kubernetes.io/projected/3c7dc27d-60b6-44e7-a7a0-f731807890a1-kube-api-access-dtzsd\") pod \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\" (UID: \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\") " Oct 29 22:35:15.206836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.194817 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7dc27d-60b6-44e7-a7a0-f731807890a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7dc27d-60b6-44e7-a7a0-f731807890a1-plugins-config-volume\") pod \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\" (UID: \"3c7dc27d-60b6-44e7-a7a0-f731807890a1\") " Oct 29 22:35:15.206836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.204659 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:15.398431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.390702 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7dc27d-60b6-44e7-a7a0-f731807890a1-plugins-config-volume" pod "3c7dc27d-60b6-44e7-a7a0-f731807890a1" (UID: "3c7dc27d-60b6-44e7-a7a0-f731807890a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:15.449420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.444196 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c7dc27d-60b6-44e7-a7a0-f731807890a1-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:15.711223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.654572 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" podUID="f937a2f3-176a-4d9d-8615-96a325e62ebb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.85:50052\" within 1s: context deadline exceeded" Oct 29 22:35:15.711223 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:35:15.655293 5610 logging.go:55] [core] [Channel #620 SubChannel #621]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.85:50052", ServerName: "10.134.0.85:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.85:50052: operation was canceled" Oct 29 22:35:15.711223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.656519 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" podUID="3c7dc27d-60b6-44e7-a7a0-f731807890a1" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.84:50052\" within 1s: context deadline exceeded" Oct 29 22:35:15.711223 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:35:15.656684 5610 logging.go:55] [core] [Channel #622 SubChannel #623]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.84:50052", ServerName: "10.134.0.84:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.84:50052: operation was canceled" Oct 29 22:35:15.820419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.819380 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7dc27d-60b6-44e7-a7a0-f731807890a1-kube-api-access-dtzsd" pod "3c7dc27d-60b6-44e7-a7a0-f731807890a1" (UID: "3c7dc27d-60b6-44e7-a7a0-f731807890a1"). InnerVolumeSpecName "kube-api-access-dtzsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:15.900714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.900680 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:35:15.901309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:15.901273 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:35:16.035635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:16.035526 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:16.093421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:16.068008 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzsd\" (UniqueName: \"kubernetes.io/projected/3c7dc27d-60b6-44e7-a7a0-f731807890a1-kube-api-access-dtzsd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:16.331474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:16.324009 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:35:16.407415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:16.404610 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 29 22:35:16.674474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:16.663114 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:35:16.896964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:16.896928 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f937a2f3-176a-4d9d-8615-96a325e62ebb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937a2f3-176a-4d9d-8615-96a325e62ebb-plugins-config-volume\") pod \"f937a2f3-176a-4d9d-8615-96a325e62ebb\" (UID: \"f937a2f3-176a-4d9d-8615-96a325e62ebb\") " Oct 29 22:35:16.910799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:16.910768 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f937a2f3-176a-4d9d-8615-96a325e62ebb-kube-api-access-dgd88\" (UniqueName: \"kubernetes.io/projected/f937a2f3-176a-4d9d-8615-96a325e62ebb-kube-api-access-dgd88\") pod \"f937a2f3-176a-4d9d-8615-96a325e62ebb\" (UID: \"f937a2f3-176a-4d9d-8615-96a325e62ebb\") " Oct 29 22:35:17.098419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.061612 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:17.098419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.062076 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f937a2f3-176a-4d9d-8615-96a325e62ebb-kube-api-access-dgd88" pod "f937a2f3-176a-4d9d-8615-96a325e62ebb" (UID: "f937a2f3-176a-4d9d-8615-96a325e62ebb"). InnerVolumeSpecName "kube-api-access-dgd88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:17.098419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.080845 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" event={"ID":"3c7dc27d-60b6-44e7-a7a0-f731807890a1","Type":"ContainerDied","Data":"86cb5129c42d92acc6458f6f1b1f6040d6344ba113861fce3284a24f76255846"} Oct 29 22:35:17.098419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.080913 5610 scope.go:122] "RemoveContainer" containerID="f3790d088fcc46a71826c8e99fe018055fe15ff828ce6465216c82f9998ae750" Oct 29 22:35:17.098419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.081250 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c" Oct 29 22:35:17.125419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.099052 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f937a2f3-176a-4d9d-8615-96a325e62ebb-plugins-config-volume" pod "f937a2f3-176a-4d9d-8615-96a325e62ebb" (UID: "f937a2f3-176a-4d9d-8615-96a325e62ebb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:17.236470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.158656 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:35:17.275433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.169488 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" event={"ID":"f937a2f3-176a-4d9d-8615-96a325e62ebb","Type":"ContainerDied","Data":"9e7c3749709f9b53773fe4f8fddb4189ed2cbf50c5df93d1b00ea88f1d0cb1dc"} Oct 29 22:35:17.275433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.169606 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph" Oct 29 22:35:17.275433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.212592 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgd88\" (UniqueName: \"kubernetes.io/projected/f937a2f3-176a-4d9d-8615-96a325e62ebb-kube-api-access-dgd88\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:17.275433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.269575 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f937a2f3-176a-4d9d-8615-96a325e62ebb-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:17.331427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.327641 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:17.406651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.376673 5610 scope.go:122] "RemoveContainer" containerID="6aac864412620c2464abe86b00ebad63437c7e42c855f8a4b61e37d90101da50" Oct 29 22:35:17.575470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.572305 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c"] Oct 29 22:35:17.605490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.605452 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1309d608-65d9d8b479-xp28c"] Oct 29 22:35:17.665557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.665516 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:17.721418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.706336 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7dc27d-60b6-44e7-a7a0-f731807890a1" path="/var/lib/kubelet/pods/3c7dc27d-60b6-44e7-a7a0-f731807890a1/volumes" Oct 29 22:35:17.742062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.742029 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph"] Oct 29 22:35:17.742285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.742270 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1309d608-6db75d7cd4-ctnph"] Oct 29 22:35:17.986421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:17.986031 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:18.099926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:18.098679 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:18.149417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:18.135226 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:18.252471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:18.241487 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:18.422523 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:18.407489 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:18.683590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:18.683480 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:18.977416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:18.976387 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:35:19.046419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:19.046153 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:19.115418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:19.111883 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:35:19.355482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:19.353467 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:35:19.424418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:19.422073 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:19.617236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:19.617154 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:35:19.700289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:19.686182 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:19.958562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:19.758021 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:20.314566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:20.222528 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:20.333427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:20.225585 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:20.403783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:20.259014 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f937a2f3-176a-4d9d-8615-96a325e62ebb" path="/var/lib/kubelet/pods/f937a2f3-176a-4d9d-8615-96a325e62ebb/volumes" Oct 29 22:35:20.448416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:20.416084 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv"] Oct 29 22:35:20.472421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:20.448729 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg"] Oct 29 22:35:20.840057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:20.840028 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:35:20.981428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:20.964366 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:20.981851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:20.966190 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:35:21.070994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:21.070938 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:21.114799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:21.112632 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:21.458461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:21.457388 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:21.597489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:21.590507 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 29 22:35:21.709474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:21.704588 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:21.922874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:21.919860 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:21.983628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:21.983516 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:35:22.070450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.063762 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:22.098267 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.098236 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:35:22.139651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.139610 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:22.140298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.140275 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:35:22.354680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.354454 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:22.457616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.457553 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:22.473582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.473538 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:35:22.679818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.679720 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv"] Oct 29 22:35:22.680283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.680249 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" podUID="af9cf790-8608-4f83-b20d-fc09ef1011f7" containerName="istio-proxy" containerID="cri-o://fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f" gracePeriod=30 Oct 29 22:35:22.929450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.923211 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:22.929450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:22.924797 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:35:23.035418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.007629 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:23.183420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.163940 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:35:23.210694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.210652 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:23.348032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.347944 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:23.467580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.467511 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:35:23.499482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.499445 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw"] Oct 29 22:35:23.520413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.501168 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f937a2f3-176a-4d9d-8615-96a325e62ebb" containerName="payload-processing" Oct 29 22:35:23.520708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.520679 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f937a2f3-176a-4d9d-8615-96a325e62ebb" containerName="payload-processing" Oct 29 22:35:23.520897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.520882 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3c7dc27d-60b6-44e7-a7a0-f731807890a1" containerName="payload-pre-processing" Oct 29 22:35:23.520987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.520975 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7dc27d-60b6-44e7-a7a0-f731807890a1" containerName="payload-pre-processing" Oct 29 22:35:23.521462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.521373 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7dc27d-60b6-44e7-a7a0-f731807890a1" containerName="payload-pre-processing" Oct 29 22:35:23.521632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.521618 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f937a2f3-176a-4d9d-8615-96a325e62ebb" containerName="payload-processing" Oct 29 22:35:23.550215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.550186 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.554432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.553778 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:35:23.643954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.643766 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw"] Oct 29 22:35:23.671463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.671365 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:23.672913 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.672823 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e1d9591d-openshift-default-dockercfg-xl6g6\"" Oct 29 22:35:23.697041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.696995 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc2f04-1165-4173-9084-8fe169540527-istiod-ca-cert\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.697423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.697385 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-data\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.709557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.709522 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-socket\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.710853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.709840 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-istio-token\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.710853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.709899 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-certs\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.710853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.709984 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc2f04-1165-4173-9084-8fe169540527-istio-podinfo\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.710853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.710033 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-credential-socket\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.710853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.710241 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-envoy\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.710853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.710517 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vlf\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-kube-api-access-56vlf\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.924675 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.813387 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc2f04-1165-4173-9084-8fe169540527-istio-podinfo\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.930534 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-credential-socket\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.930588 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-envoy\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.930844 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vlf\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-kube-api-access-56vlf\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.930977 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc2f04-1165-4173-9084-8fe169540527-istiod-ca-cert\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.931115 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-data\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.931207 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-socket\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.931251 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-istio-token\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.931290 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-certs\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.952519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.931930 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-certs\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:23.993440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:23.992006 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-credential-socket\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.073542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.072615 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc2f04-1165-4173-9084-8fe169540527-istiod-ca-cert\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.081661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.081387 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:24.094882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.094847 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-socket\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.095528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.095499 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-data\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.134944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.134904 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-envoy\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.218193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.216780 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:24.236440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.234744 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:24.298419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.280546 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc2f04-1165-4173-9084-8fe169540527-istio-podinfo\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.298419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.281024 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vlf\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-kube-api-access-56vlf\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.298419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.281102 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:35:24.438276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.438244 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-istio-token\") pod \"e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.440053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.440026 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:24.744256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.744226 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:35:24.746482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.746453 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:35:24.747565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:24.747542 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:35:25.411813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.411761 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:25.412095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.412071 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:25.412301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.412276 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:25.441690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.441660 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw"] Oct 29 22:35:25.484414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.482730 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" podUID="bb22f6e9-4827-4e18-942a-458f7073f63c" containerName="payload-pre-processing" containerID="cri-o://a84f21561e82b8d93a06cb64064e2fa97a42ac39025dd7c99003c1d7e1d6e788" gracePeriod=25 Oct 29 22:35:25.484414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.482990 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" podUID="d65a0c50-138d-48dc-94f8-ff440f010f56" containerName="payload-processing" containerID="cri-o://27fd3fecb6d81949507430ed2482251d17b7dc109540f20468f86a6c5497b402" gracePeriod=25 Oct 29 22:35:25.721340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.721302 5610 generic.go:356] "Generic (PLEG): container finished" podID="d65a0c50-138d-48dc-94f8-ff440f010f56" containerID="27fd3fecb6d81949507430ed2482251d17b7dc109540f20468f86a6c5497b402" exitCode=0 Oct 29 22:35:25.721617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.721375 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" event={"ID":"d65a0c50-138d-48dc-94f8-ff440f010f56","Type":"ContainerDied","Data":"27fd3fecb6d81949507430ed2482251d17b7dc109540f20468f86a6c5497b402"} Oct 29 22:35:25.725812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.725777 5610 generic.go:356] "Generic (PLEG): container finished" podID="bb22f6e9-4827-4e18-942a-458f7073f63c" containerID="a84f21561e82b8d93a06cb64064e2fa97a42ac39025dd7c99003c1d7e1d6e788" exitCode=0 Oct 29 22:35:25.725986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.725914 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" event={"ID":"bb22f6e9-4827-4e18-942a-458f7073f63c","Type":"ContainerDied","Data":"a84f21561e82b8d93a06cb64064e2fa97a42ac39025dd7c99003c1d7e1d6e788"} Oct 29 22:35:25.732319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.730957 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" event={"ID":"0ebc2f04-1165-4173-9084-8fe169540527","Type":"ContainerStarted","Data":"d0b0880c42e09b0787d38fcdb290dc21d01fc3dc05699be3f48935d01521b354"} Oct 29 22:35:25.732319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.731018 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" event={"ID":"0ebc2f04-1165-4173-9084-8fe169540527","Type":"ContainerStarted","Data":"6f9a07323c0884baa0324391de40da89ac0e468594160e917afe8948a33a0c46"} Oct 29 22:35:25.828489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.828431 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" podStartSLOduration=2.82841252 podStartE2EDuration="2.82841252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:25.826942984 +0000 UTC m=+1188.308260483" watchObservedRunningTime="2026-10-29 22:35:25.82841252 +0000 UTC m=+1188.309730016" Oct 29 22:35:25.833350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:25.832953 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:26.212428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.212336 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:26.474882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.474539 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:35:26.562373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.562019 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:35:26.635501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.635457 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65a0c50-138d-48dc-94f8-ff440f010f56-kube-api-access-z4hdz\" (UniqueName: \"kubernetes.io/projected/d65a0c50-138d-48dc-94f8-ff440f010f56-kube-api-access-z4hdz\") pod \"d65a0c50-138d-48dc-94f8-ff440f010f56\" (UID: \"d65a0c50-138d-48dc-94f8-ff440f010f56\") " Oct 29 22:35:26.635683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.635661 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65a0c50-138d-48dc-94f8-ff440f010f56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65a0c50-138d-48dc-94f8-ff440f010f56-plugins-config-volume\") pod \"d65a0c50-138d-48dc-94f8-ff440f010f56\" (UID: \"d65a0c50-138d-48dc-94f8-ff440f010f56\") " Oct 29 22:35:26.635767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.635712 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb22f6e9-4827-4e18-942a-458f7073f63c-kube-api-access-sxqj6\" (UniqueName: \"kubernetes.io/projected/bb22f6e9-4827-4e18-942a-458f7073f63c-kube-api-access-sxqj6\") pod \"bb22f6e9-4827-4e18-942a-458f7073f63c\" (UID: \"bb22f6e9-4827-4e18-942a-458f7073f63c\") " Oct 29 22:35:26.635767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.635760 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb22f6e9-4827-4e18-942a-458f7073f63c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb22f6e9-4827-4e18-942a-458f7073f63c-plugins-config-volume\") pod \"bb22f6e9-4827-4e18-942a-458f7073f63c\" (UID: \"bb22f6e9-4827-4e18-942a-458f7073f63c\") " Oct 29 22:35:26.636322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.636297 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65a0c50-138d-48dc-94f8-ff440f010f56-plugins-config-volume" pod "d65a0c50-138d-48dc-94f8-ff440f010f56" (UID: "d65a0c50-138d-48dc-94f8-ff440f010f56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:26.637789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.637760 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb22f6e9-4827-4e18-942a-458f7073f63c-plugins-config-volume" pod "bb22f6e9-4827-4e18-942a-458f7073f63c" (UID: "bb22f6e9-4827-4e18-942a-458f7073f63c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:26.639361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.639334 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65a0c50-138d-48dc-94f8-ff440f010f56-kube-api-access-z4hdz" pod "d65a0c50-138d-48dc-94f8-ff440f010f56" (UID: "d65a0c50-138d-48dc-94f8-ff440f010f56"). InnerVolumeSpecName "kube-api-access-z4hdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:26.639830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.639801 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb22f6e9-4827-4e18-942a-458f7073f63c-kube-api-access-sxqj6" pod "bb22f6e9-4827-4e18-942a-458f7073f63c" (UID: "bb22f6e9-4827-4e18-942a-458f7073f63c"). InnerVolumeSpecName "kube-api-access-sxqj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:26.652800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.652583 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:26.748462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.744835 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d65a0c50-138d-48dc-94f8-ff440f010f56-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:26.748462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.744870 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxqj6\" (UniqueName: \"kubernetes.io/projected/bb22f6e9-4827-4e18-942a-458f7073f63c-kube-api-access-sxqj6\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:26.748462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.744887 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb22f6e9-4827-4e18-942a-458f7073f63c-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:26.748462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.744905 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4hdz\" (UniqueName: \"kubernetes.io/projected/d65a0c50-138d-48dc-94f8-ff440f010f56-kube-api-access-z4hdz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:26.880478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.872185 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" event={"ID":"d65a0c50-138d-48dc-94f8-ff440f010f56","Type":"ContainerDied","Data":"be66d28535e0a133c4da6eecc7e9d411070db92282fbb6cb65b731dda118269a"} Oct 29 22:35:26.880478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.872263 5610 scope.go:122] "RemoveContainer" containerID="27fd3fecb6d81949507430ed2482251d17b7dc109540f20468f86a6c5497b402" Oct 29 22:35:26.880478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.872466 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv" Oct 29 22:35:26.969513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.907409 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:26.969513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.936904 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" Oct 29 22:35:26.969513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.953681 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg" event={"ID":"bb22f6e9-4827-4e18-942a-458f7073f63c","Type":"ContainerDied","Data":"a0464d94d0071e7fe738390c0937ecd2c7d2c73483090225716616cf0ddcf9e5"} Oct 29 22:35:26.969513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.953759 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:26.975828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:26.975755 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:35:27.153940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.153810 5610 scope.go:122] "RemoveContainer" containerID="a84f21561e82b8d93a06cb64064e2fa97a42ac39025dd7c99003c1d7e1d6e788" Oct 29 22:35:27.206092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.206024 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv"] Oct 29 22:35:27.227951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.227910 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-3593b8ba-66df9dddf4-5rbrv"] Oct 29 22:35:27.237594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.236984 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg"] Oct 29 22:35:27.246310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.246262 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3593b8ba-b5c889f7f-fr5fg"] Oct 29 22:35:27.626938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.626893 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:27.644110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.644067 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb22f6e9-4827-4e18-942a-458f7073f63c" path="/var/lib/kubelet/pods/bb22f6e9-4827-4e18-942a-458f7073f63c/volumes" Oct 29 22:35:27.644830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.644807 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65a0c50-138d-48dc-94f8-ff440f010f56" path="/var/lib/kubelet/pods/d65a0c50-138d-48dc-94f8-ff440f010f56/volumes" Oct 29 22:35:27.779051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:27.779014 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:28.126514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:28.126479 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:29.331668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.331569 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:29.473467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.473430 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt"] Oct 29 22:35:29.474269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.474246 5610 cpu_manager.go:414] "containerMap: removing container" podUID="bb22f6e9-4827-4e18-942a-458f7073f63c" containerName="payload-pre-processing" Oct 29 22:35:29.474482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.474269 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb22f6e9-4827-4e18-942a-458f7073f63c" containerName="payload-pre-processing" Oct 29 22:35:29.474482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.474324 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d65a0c50-138d-48dc-94f8-ff440f010f56" containerName="payload-processing" Oct 29 22:35:29.474482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.474332 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65a0c50-138d-48dc-94f8-ff440f010f56" containerName="payload-processing" Oct 29 22:35:29.474701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.474580 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb22f6e9-4827-4e18-942a-458f7073f63c" containerName="payload-pre-processing" Oct 29 22:35:29.474701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.474609 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65a0c50-138d-48dc-94f8-ff440f010f56" containerName="payload-processing" Oct 29 22:35:29.480561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.480532 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:29.482329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.482305 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:29.510859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.510824 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt"] Oct 29 22:35:29.544364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.536828 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e1d9591d-dockercfg-s6lqg\"" Oct 29 22:35:29.555446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.548469 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e1d9591d\"" Oct 29 22:35:29.782416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.769705 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93721f18-92c5-42ff-aca9-29b79893e6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt\" (UID: \"93721f18-92c5-42ff-aca9-29b79893e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:29.782416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.769819 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25cph\" (UniqueName: \"kubernetes.io/projected/93721f18-92c5-42ff-aca9-29b79893e6d3-kube-api-access-25cph\") pod \"payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt\" (UID: \"93721f18-92c5-42ff-aca9-29b79893e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:29.876924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.876850 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93721f18-92c5-42ff-aca9-29b79893e6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt\" (UID: \"93721f18-92c5-42ff-aca9-29b79893e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:29.877133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.876996 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25cph\" (UniqueName: \"kubernetes.io/projected/93721f18-92c5-42ff-aca9-29b79893e6d3-kube-api-access-25cph\") pod \"payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt\" (UID: \"93721f18-92c5-42ff-aca9-29b79893e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:29.886442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.878076 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93721f18-92c5-42ff-aca9-29b79893e6d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt\" (UID: \"93721f18-92c5-42ff-aca9-29b79893e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:29.946487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:29.945363 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25cph\" (UniqueName: \"kubernetes.io/projected/93721f18-92c5-42ff-aca9-29b79893e6d3-kube-api-access-25cph\") pod \"payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt\" (UID: \"93721f18-92c5-42ff-aca9-29b79893e6d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:30.124664 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:30.124218 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:30.287830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:30.286947 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:30.506622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:30.506590 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt"] Oct 29 22:35:30.507773 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:35:30.507740 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93721f18_92c5_42ff_aca9_29b79893e6d3.slice/crio-7f3b32db758e4dcf98895ef7fccfe85112b169755be11288acc7a5bab22c0338 WatchSource:0}: Error finding container 7f3b32db758e4dcf98895ef7fccfe85112b169755be11288acc7a5bab22c0338: Status 404 returned error can't find the container with id 7f3b32db758e4dcf98895ef7fccfe85112b169755be11288acc7a5bab22c0338 Oct 29 22:35:30.564570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:30.564534 5610 patch_prober.go:29] interesting pod/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:35:30.564883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:30.564596 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" podUID="af9cf790-8608-4f83-b20d-fc09ef1011f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:35:30.996303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:30.996262 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" event={"ID":"93721f18-92c5-42ff-aca9-29b79893e6d3","Type":"ContainerStarted","Data":"87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406"} Oct 29 22:35:30.996546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:30.996325 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" event={"ID":"93721f18-92c5-42ff-aca9-29b79893e6d3","Type":"ContainerStarted","Data":"7f3b32db758e4dcf98895ef7fccfe85112b169755be11288acc7a5bab22c0338"} Oct 29 22:35:30.996546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:30.996354 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:31.038281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:31.038204 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" podStartSLOduration=2.038185246 podStartE2EDuration="2.038185246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:31.035813093 +0000 UTC m=+1193.517130582" watchObservedRunningTime="2026-10-29 22:35:31.038185246 +0000 UTC m=+1193.519502741" Oct 29 22:35:32.834247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:32.829765 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:33.692082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.692054 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:35:33.853737 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.853701 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-certs\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.853824 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-podinfo\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.853877 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-envoy\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.853916 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-data\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.853954 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-certs" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:33.854133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854036 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-credential-socket\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854165 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-token\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-token\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854208 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-kube-api-access-8xsgz\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-kube-api-access-8xsgz\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854268 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9cf790-8608-4f83-b20d-fc09ef1011f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9cf790-8608-4f83-b20d-fc09ef1011f7-istiod-ca-cert\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854301 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-socket\") pod \"af9cf790-8608-4f83-b20d-fc09ef1011f7\" (UID: \"af9cf790-8608-4f83-b20d-fc09ef1011f7\") " Oct 29 22:35:33.854710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854297 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-credential-socket" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:33.854767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854532 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-data" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:33.854767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854622 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9cf790-8608-4f83-b20d-fc09ef1011f7-istiod-ca-cert" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:33.855233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.854942 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-socket" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:33.855233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.855171 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:33.855233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.855188 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:33.855233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.855199 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9cf790-8608-4f83-b20d-fc09ef1011f7-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:33.855233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.855208 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:33.855233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.855216 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:33.856547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.856519 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-token" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:33.856738 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.856717 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-podinfo" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:35:33.856797 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.856786 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-envoy" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:33.856905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.856892 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-kube-api-access-8xsgz" pod "af9cf790-8608-4f83-b20d-fc09ef1011f7" (UID: "af9cf790-8608-4f83-b20d-fc09ef1011f7"). InnerVolumeSpecName "kube-api-access-8xsgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:33.956705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.956668 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:33.956705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.956708 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xsgz\" (UniqueName: \"kubernetes.io/projected/af9cf790-8608-4f83-b20d-fc09ef1011f7-kube-api-access-8xsgz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:33.956924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.956724 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:33.956924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:33.956739 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9cf790-8608-4f83-b20d-fc09ef1011f7-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:34.236104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.236000 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:34.290718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.290681 5610 generic.go:356] "Generic (PLEG): container finished" podID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerID="67a51e8f740f07ebcc4ecd69c8d30880e12423407be72f46d8d187d6f563e399" exitCode=0 Oct 29 22:35:34.290888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.290753 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" event={"ID":"43bce1d9-bceb-4c3a-9cc5-67de580eace1","Type":"ContainerDied","Data":"67a51e8f740f07ebcc4ecd69c8d30880e12423407be72f46d8d187d6f563e399"} Oct 29 22:35:34.291281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.291264 5610 scope.go:122] "RemoveContainer" containerID="67a51e8f740f07ebcc4ecd69c8d30880e12423407be72f46d8d187d6f563e399" Oct 29 22:35:34.293196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.293168 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:34.293304 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.293231 5610 generic.go:356] "Generic (PLEG): container finished" podID="af9cf790-8608-4f83-b20d-fc09ef1011f7" containerID="fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f" exitCode=0 Oct 29 22:35:34.293304 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.293254 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" event={"ID":"af9cf790-8608-4f83-b20d-fc09ef1011f7","Type":"ContainerDied","Data":"fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f"} Oct 29 22:35:34.293304 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.293283 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" Oct 29 22:35:34.293304 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.293296 5610 scope.go:122] "RemoveContainer" containerID="fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f" Oct 29 22:35:34.293304 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.293241 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:34.293546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.293361 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:34.293546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.293288 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv" event={"ID":"af9cf790-8608-4f83-b20d-fc09ef1011f7","Type":"ContainerDied","Data":"b95affc1f3611f2719d98beea0966886fe6ba04ddc18e178ac6436f36ff9f3bd"} Oct 29 22:35:34.311792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.311771 5610 scope.go:122] "RemoveContainer" containerID="fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f" Oct 29 22:35:34.312122 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:35:34.312097 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f\": container with ID starting with fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f not found: ID does not exist" containerID="fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f" Oct 29 22:35:34.312198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.312135 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f"} err="failed to get container status \"fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f\": rpc error: code = NotFound desc = could not find container \"fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f\": container with ID starting with fe8bb40789f6f7f846835d37e96ee6b031cc68c24e7f221aa44d6c56f098b82f not found: ID does not exist" Oct 29 22:35:34.695576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.695473 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv"] Oct 29 22:35:34.720113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.720074 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1309d608-openshift-default-cfc7dfc5-4nnqv"] Oct 29 22:35:34.900278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:34.900189 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:35.033260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:35.033222 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:35.201541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:35.196230 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n"] Oct 29 22:35:35.231378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:35.231284 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:35.349362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:35.348702 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" event={"ID":"43bce1d9-bceb-4c3a-9cc5-67de580eace1","Type":"ContainerStarted","Data":"62458c7b88fb08aa13dffc7c89b30048913fc755c964f0f5b98489fa7d420ec7"} Oct 29 22:35:35.349362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:35.348742 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" podUID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerName="istio-proxy" containerID="cri-o://62458c7b88fb08aa13dffc7c89b30048913fc755c964f0f5b98489fa7d420ec7" gracePeriod=30 Oct 29 22:35:35.683487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:35.678578 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9cf790-8608-4f83-b20d-fc09ef1011f7" path="/var/lib/kubelet/pods/af9cf790-8608-4f83-b20d-fc09ef1011f7/volumes" Oct 29 22:35:36.153128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:36.153084 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:36.404799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:36.404709 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:36.724343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:36.724310 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:36.859143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:36.859114 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:37.179431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:37.179333 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:37.347634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:37.323941 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:37.784462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:37.784421 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:35:38.935177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:38.935135 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:40.430899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.420277 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:40.482582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.471938 5610 generic.go:356] "Generic (PLEG): container finished" podID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerID="62458c7b88fb08aa13dffc7c89b30048913fc755c964f0f5b98489fa7d420ec7" exitCode=0 Oct 29 22:35:40.482582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.471994 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" event={"ID":"43bce1d9-bceb-4c3a-9cc5-67de580eace1","Type":"ContainerDied","Data":"62458c7b88fb08aa13dffc7c89b30048913fc755c964f0f5b98489fa7d420ec7"} Oct 29 22:35:40.482582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.472325 5610 scope.go:122] "RemoveContainer" containerID="67a51e8f740f07ebcc4ecd69c8d30880e12423407be72f46d8d187d6f563e399" Oct 29 22:35:40.819614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.819587 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:35:40.871657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.871621 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-data\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.871829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.871732 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-kube-api-access-2fkh4\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-kube-api-access-2fkh4\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.871829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.871771 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-credential-socket\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.871829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.871811 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-envoy\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.872063 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872040 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istiod-ca-cert\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.872157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872086 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-socket\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.872157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872120 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-token\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-token\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.872301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872249 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-certs\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.872360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872305 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-credential-socket" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.872360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872313 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-podinfo\") pod \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\" (UID: \"43bce1d9-bceb-4c3a-9cc5-67de580eace1\") " Oct 29 22:35:40.872672 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872644 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-socket" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.872845 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872822 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-data" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.872990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.872960 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-certs" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.873108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.873084 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istiod-ca-cert" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:40.873515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.873368 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.875043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.875004 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.875174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.875160 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.875283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.875271 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.875401 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.875363 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-podinfo" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:35:40.875503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.875386 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-envoy" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:35:40.875560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.875544 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-token" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:40.875636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.875620 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.876976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.876955 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-kube-api-access-2fkh4" pod "43bce1d9-bceb-4c3a-9cc5-67de580eace1" (UID: "43bce1d9-bceb-4c3a-9cc5-67de580eace1"). InnerVolumeSpecName "kube-api-access-2fkh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:40.977512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.977478 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.977512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.977509 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.977735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.977539 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fkh4\" (UniqueName: \"kubernetes.io/projected/43bce1d9-bceb-4c3a-9cc5-67de580eace1-kube-api-access-2fkh4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:40.977735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:40.977550 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43bce1d9-bceb-4c3a-9cc5-67de580eace1-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:41.484588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:41.484555 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" Oct 29 22:35:41.485055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:41.484562 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n" event={"ID":"43bce1d9-bceb-4c3a-9cc5-67de580eace1","Type":"ContainerDied","Data":"59be70b8b347f94473b7fd1c493113b028f7bcf69b5e952c2edea1ace2f9aad4"} Oct 29 22:35:41.485055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:41.484717 5610 scope.go:122] "RemoveContainer" containerID="62458c7b88fb08aa13dffc7c89b30048913fc755c964f0f5b98489fa7d420ec7" Oct 29 22:35:41.549289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:41.549257 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n"] Oct 29 22:35:41.552922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:41.552886 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3593b8ba-gw-openshift-default-594887f8df-btd2n"] Oct 29 22:35:41.637381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:41.637340 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" path="/var/lib/kubelet/pods/43bce1d9-bceb-4c3a-9cc5-67de580eace1/volumes" Oct 29 22:35:41.877918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:41.877824 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:42.274032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:42.274004 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:35:43.582418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:43.576930 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:43.786442 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:35:43.784066 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:35:45.150822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:45.150782 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:46.657715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.657678 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg"] Oct 29 22:35:46.658464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658447 5610 cpu_manager.go:414] "containerMap: removing container" podUID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerName="istio-proxy" Oct 29 22:35:46.658539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658465 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerName="istio-proxy" Oct 29 22:35:46.658539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658475 5610 cpu_manager.go:414] "containerMap: removing container" podUID="af9cf790-8608-4f83-b20d-fc09ef1011f7" containerName="istio-proxy" Oct 29 22:35:46.658539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658481 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9cf790-8608-4f83-b20d-fc09ef1011f7" containerName="istio-proxy" Oct 29 22:35:46.658539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658498 5610 cpu_manager.go:414] "containerMap: removing container" podUID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerName="istio-proxy" Oct 29 22:35:46.658539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658503 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerName="istio-proxy" Oct 29 22:35:46.658703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658632 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerName="istio-proxy" Oct 29 22:35:46.658703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658641 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9cf790-8608-4f83-b20d-fc09ef1011f7" containerName="istio-proxy" Oct 29 22:35:46.658703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.658651 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bce1d9-bceb-4c3a-9cc5-67de580eace1" containerName="istio-proxy" Oct 29 22:35:46.662227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.662206 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.691778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.691745 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e1d9591d-serving-cert\"" Oct 29 22:35:46.696937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.696907 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg"] Oct 29 22:35:46.734075 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.734037 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-metrics-certs\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.734306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.734283 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmm2t\" (UniqueName: \"kubernetes.io/projected/c1617725-7d72-4a55-9084-e541e08ac95e-kube-api-access-dmm2t\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.734368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.734350 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1617725-7d72-4a55-9084-e541e08ac95e-openshift-service-ca\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.734483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.734464 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-maas-api-tls\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.784965 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.784928 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:46.835882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.835848 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmm2t\" (UniqueName: \"kubernetes.io/projected/c1617725-7d72-4a55-9084-e541e08ac95e-kube-api-access-dmm2t\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.836053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.835905 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1617725-7d72-4a55-9084-e541e08ac95e-openshift-service-ca\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.836053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.835939 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-maas-api-tls\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.836125 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.836064 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-metrics-certs\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.836703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.836673 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1617725-7d72-4a55-9084-e541e08ac95e-openshift-service-ca\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.838622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.838602 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-metrics-certs\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.838693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.838611 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-maas-api-tls\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.854295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.854271 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmm2t\" (UniqueName: \"kubernetes.io/projected/c1617725-7d72-4a55-9084-e541e08ac95e-kube-api-access-dmm2t\") pod \"maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:46.927571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.918948 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:46.987853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:46.987822 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:47.065265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:47.059329 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:47.250995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:47.250936 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:47.401269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:47.401215 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:47.589224 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:35:47.589198 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1617725_7d72_4a55_9084_e541e08ac95e.slice/crio-878a5f45937a56f2aa6eae489c3c73355be1462c9d318ba0debd3ef7cfac6d72 WatchSource:0}: Error finding container 878a5f45937a56f2aa6eae489c3c73355be1462c9d318ba0debd3ef7cfac6d72: Status 404 returned error can't find the container with id 878a5f45937a56f2aa6eae489c3c73355be1462c9d318ba0debd3ef7cfac6d72 Oct 29 22:35:47.597019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:47.596906 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg"] Oct 29 22:35:47.890972 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:47.890847 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:48.034454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:48.034357 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:48.642217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:48.642177 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" event={"ID":"c1617725-7d72-4a55-9084-e541e08ac95e","Type":"ContainerStarted","Data":"e8f324ff6bd2c3be61141542ce0725255233327bbda504aa47dae088d2139a94"} Oct 29 22:35:48.642422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:48.642263 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:48.642422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:48.642291 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" event={"ID":"c1617725-7d72-4a55-9084-e541e08ac95e","Type":"ContainerStarted","Data":"878a5f45937a56f2aa6eae489c3c73355be1462c9d318ba0debd3ef7cfac6d72"} Oct 29 22:35:48.684985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:48.684922 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" podStartSLOduration=2.157096368 podStartE2EDuration="2.684903055s" totalImagesPullingTime="527.806687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:48.681449562 +0000 UTC m=+1211.162767056" watchObservedRunningTime="2026-10-29 22:35:48.684903055 +0000 UTC m=+1211.166220552" Oct 29 22:35:48.938691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:48.938606 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:50.486671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:50.486604 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98"] Oct 29 22:35:50.487185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:50.486866 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" podUID="b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" containerName="maas-api" containerID="cri-o://35210326513fa37d4bda9f0f201349536153071676176d8b885180c47d127e5b" gracePeriod=30 Oct 29 22:35:50.592054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:50.592022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:50.658971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:50.658939 5610 generic.go:356] "Generic (PLEG): container finished" podID="b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" containerID="35210326513fa37d4bda9f0f201349536153071676176d8b885180c47d127e5b" exitCode=0 Oct 29 22:35:50.659134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:50.659010 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" event={"ID":"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa","Type":"ContainerDied","Data":"35210326513fa37d4bda9f0f201349536153071676176d8b885180c47d127e5b"} Oct 29 22:35:51.141908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.141874 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:35:51.246426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.244983 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-metrics-certs\") pod \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " Oct 29 22:35:51.246426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.245228 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-kube-api-access-czl89\" (UniqueName: \"kubernetes.io/projected/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-kube-api-access-czl89\") pod \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " Oct 29 22:35:51.246426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.245276 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls\") pod \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " Oct 29 22:35:51.246426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.245431 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-openshift-service-ca\") pod \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\" (UID: \"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa\") " Oct 29 22:35:51.249654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.247615 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-openshift-service-ca" pod "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" (UID: "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:35:51.280476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.276790 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-kube-api-access-czl89" pod "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" (UID: "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa"). InnerVolumeSpecName "kube-api-access-czl89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:35:51.283248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.283107 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-metrics-certs" pod "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" (UID: "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:51.283248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.283210 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls" pod "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" (UID: "b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:35:51.379490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.359058 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czl89\" (UniqueName: \"kubernetes.io/projected/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-kube-api-access-czl89\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:51.379490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.359099 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:51.379490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.359115 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:51.379490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.359128 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:35:51.565437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.565324 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:51.669197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.669164 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" Oct 29 22:35:51.669358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.669198 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98" event={"ID":"b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa","Type":"ContainerDied","Data":"1e905f7454d631fbe282518ed7f1b5bb48ee1661484c0a67af90e2c76cdab18f"} Oct 29 22:35:51.669358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.669259 5610 scope.go:122] "RemoveContainer" containerID="35210326513fa37d4bda9f0f201349536153071676176d8b885180c47d127e5b" Oct 29 22:35:51.732993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.732953 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98"] Oct 29 22:35:51.735765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.735736 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:51.744343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.744319 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-591986bb-9d44555f7-rnw98"] Oct 29 22:35:51.898187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:51.898090 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:52.109968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:52.109933 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:53.291463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:53.291379 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:53.655468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:53.652128 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:53.728012 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:53.726802 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" path="/var/lib/kubelet/pods/b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa/volumes" Oct 29 22:35:53.951111 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:53.951076 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:54.421989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.421898 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5"] Oct 29 22:35:54.423115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.423089 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" containerName="maas-api" Oct 29 22:35:54.423232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.423147 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" containerName="maas-api" Oct 29 22:35:54.423558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.423536 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b6e8e8-a3b5-4fab-8a75-597dedc3dafa" containerName="maas-api" Oct 29 22:35:54.428938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.428918 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.444592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.444569 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-36b46d-openshift-default-dockercfg-9qfvs\"" Oct 29 22:35:54.461119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.461080 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5"] Oct 29 22:35:54.508069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.508031 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:54.625460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.625407 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55441a0e-ab81-44df-8301-956c67fecc3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.625653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.625483 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.625653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.625528 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.625653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.625587 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.625653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.625631 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.625865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.625782 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2jp\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-kube-api-access-jn2jp\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.625865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.625843 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.666419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.625997 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55441a0e-ab81-44df-8301-956c67fecc3b-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.666419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.626085 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.734422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.734024 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.737530 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.737596 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.737644 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.737773 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2jp\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-kube-api-access-jn2jp\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.737850 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.737972 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55441a0e-ab81-44df-8301-956c67fecc3b-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.738047 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.738095 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55441a0e-ab81-44df-8301-956c67fecc3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.743421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.738139 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.754248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.752149 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.754248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.752443 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-data\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.785422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.776000 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-credential-socket\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.785422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.776727 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55441a0e-ab81-44df-8301-956c67fecc3b-istiod-ca-cert\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.785422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.783427 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:35:54.800187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.795750 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-certs\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.856530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.837682 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-envoy\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.882492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.880304 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2jp\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-kube-api-access-jn2jp\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.887118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.886960 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-istio-token\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:54.918832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:54.918800 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55441a0e-ab81-44df-8301-956c67fecc3b-istio-podinfo\") pod \"e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:55.064109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:55.064024 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:55.527722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:55.527693 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5"] Oct 29 22:35:55.528160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:35:55.528132 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55441a0e_ab81_44df_8301_956c67fecc3b.slice/crio-6a6351277ce043c7d38f702c408b2e62dc2fd12287dbed3003c95685b424c06f WatchSource:0}: Error finding container 6a6351277ce043c7d38f702c408b2e62dc2fd12287dbed3003c95685b424c06f: Status 404 returned error can't find the container with id 6a6351277ce043c7d38f702c408b2e62dc2fd12287dbed3003c95685b424c06f Oct 29 22:35:55.530563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:55.530533 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:55.530628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:55.530594 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:55.530628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:55.530622 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:35:55.837344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:55.837206 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:55.920265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:55.920203 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" event={"ID":"55441a0e-ab81-44df-8301-956c67fecc3b","Type":"ContainerStarted","Data":"1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57"} Oct 29 22:35:55.920539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:55.920516 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" event={"ID":"55441a0e-ab81-44df-8301-956c67fecc3b","Type":"ContainerStarted","Data":"6a6351277ce043c7d38f702c408b2e62dc2fd12287dbed3003c95685b424c06f"} Oct 29 22:35:56.002029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:56.001306 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podStartSLOduration=2.001290166 podStartE2EDuration="2.001290166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:35:55.978164883 +0000 UTC m=+1218.459482379" watchObservedRunningTime="2026-10-29 22:35:56.001290166 +0000 UTC m=+1218.482607658" Oct 29 22:35:56.069749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:56.069721 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:56.070587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:56.070557 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:35:56.070704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:56.070616 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 29 22:35:57.070143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:57.070113 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:57.374686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:57.374586 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:57.606610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:57.606502 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:57.748966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:57.748931 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:57.933778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:57.933742 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:57.934835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:57.934809 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:35:58.440601 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:58.440569 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:58.601288 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:58.601254 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:35:59.261768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:35:59.261718 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:00.131576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:00.131543 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:00.284229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:00.284197 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:00.580999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:00.580813 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:01.638072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:01.638033 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:02.488418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:02.486382 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:03.554009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:03.553962 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:05.041546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:05.041515 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:06.833610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:06.833532 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:08.500533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:08.500497 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:10.249081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:10.249038 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:11.766531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:11.763767 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:12.590522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:12.590489 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:13.671543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:13.656153 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:13.853739 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:36:13.853705 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:36:13.969774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:13.969741 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:15.140844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.140813 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq"] Oct 29 22:36:15.145785 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.145764 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.174053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.174022 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-36b46d-serving-cert\"" Oct 29 22:36:15.178282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.178255 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq"] Oct 29 22:36:15.254366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.254330 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.254571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.254534 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.254661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.254644 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdkst\" (UniqueName: \"kubernetes.io/projected/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-kube-api-access-jdkst\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.254731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.254719 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.268594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.268554 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:15.355987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.355954 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.356171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.356107 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.356238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.356227 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.356326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.356313 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdkst\" (UniqueName: \"kubernetes.io/projected/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-kube-api-access-jdkst\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.356883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.356858 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.358368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.358345 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.358537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.358520 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.412259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.402139 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdkst\" (UniqueName: \"kubernetes.io/projected/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-kube-api-access-jdkst\") pod \"maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.426420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.420050 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:15.465429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.462533 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:15.918918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:15.918891 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq"] Oct 29 22:36:15.920274 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:36:15.920250 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c6778f_8a94_4a33_bc9e_2ed413fd0d70.slice/crio-b7a1b0db746b2d2f0953de643000f4213eb847ed29fbfd538d7d0010623705d0 WatchSource:0}: Error finding container b7a1b0db746b2d2f0953de643000f4213eb847ed29fbfd538d7d0010623705d0: Status 404 returned error can't find the container with id b7a1b0db746b2d2f0953de643000f4213eb847ed29fbfd538d7d0010623705d0 Oct 29 22:36:16.085769 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:16.085728 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" event={"ID":"60c6778f-8a94-4a33-bc9e-2ed413fd0d70","Type":"ContainerStarted","Data":"b7a1b0db746b2d2f0953de643000f4213eb847ed29fbfd538d7d0010623705d0"} Oct 29 22:36:17.098231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:17.098180 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" event={"ID":"60c6778f-8a94-4a33-bc9e-2ed413fd0d70","Type":"ContainerStarted","Data":"0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a"} Oct 29 22:36:17.098767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:17.098326 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:17.139707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:17.139648 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" podStartSLOduration=1.748911042 podStartE2EDuration="2.139632366s" totalImagesPullingTime="390.721324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:17.137556158 +0000 UTC m=+1239.618873652" watchObservedRunningTime="2026-10-29 22:36:17.139632366 +0000 UTC m=+1239.620949899" Oct 29 22:36:17.338208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:17.338132 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:19.350928 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:19.350897 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:19.497949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:19.497880 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:20.303758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:20.303719 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:21.025042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:21.024960 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:22.541480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:22.541438 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:23.112891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:23.112856 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 22:36:24.028899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:24.028857 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:24.806720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:24.806681 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt"] Oct 29 22:36:24.954679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:24.954606 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:25.092061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:25.091978 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:25.476420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:25.475898 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg"] Oct 29 22:36:25.476420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:25.476188 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" podUID="c1617725-7d72-4a55-9084-e541e08ac95e" containerName="maas-api" containerID="cri-o://e8f324ff6bd2c3be61141542ce0725255233327bbda504aa47dae088d2139a94" gracePeriod=30 Oct 29 22:36:25.530727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:25.530681 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:26.416417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.410348 5610 generic.go:356] "Generic (PLEG): container finished" podID="c1617725-7d72-4a55-9084-e541e08ac95e" containerID="e8f324ff6bd2c3be61141542ce0725255233327bbda504aa47dae088d2139a94" exitCode=0 Oct 29 22:36:26.416417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.410505 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" event={"ID":"c1617725-7d72-4a55-9084-e541e08ac95e","Type":"ContainerDied","Data":"e8f324ff6bd2c3be61141542ce0725255233327bbda504aa47dae088d2139a94"} Oct 29 22:36:26.873451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.866447 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:26.934020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.932624 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:36:26.957086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.956168 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-metrics-certs\") pod \"c1617725-7d72-4a55-9084-e541e08ac95e\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " Oct 29 22:36:26.957086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.956422 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1617725-7d72-4a55-9084-e541e08ac95e-kube-api-access-dmm2t\" (UniqueName: \"kubernetes.io/projected/c1617725-7d72-4a55-9084-e541e08ac95e-kube-api-access-dmm2t\") pod \"c1617725-7d72-4a55-9084-e541e08ac95e\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " Oct 29 22:36:26.957086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.956499 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-maas-api-tls\") pod \"c1617725-7d72-4a55-9084-e541e08ac95e\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " Oct 29 22:36:26.957086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.956599 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1617725-7d72-4a55-9084-e541e08ac95e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1617725-7d72-4a55-9084-e541e08ac95e-openshift-service-ca\") pod \"c1617725-7d72-4a55-9084-e541e08ac95e\" (UID: \"c1617725-7d72-4a55-9084-e541e08ac95e\") " Oct 29 22:36:26.959006 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:26.958958 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1617725-7d72-4a55-9084-e541e08ac95e-openshift-service-ca" pod "c1617725-7d72-4a55-9084-e541e08ac95e" (UID: "c1617725-7d72-4a55-9084-e541e08ac95e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:27.009677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.009600 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-metrics-certs" pod "c1617725-7d72-4a55-9084-e541e08ac95e" (UID: "c1617725-7d72-4a55-9084-e541e08ac95e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:27.009933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.009903 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-maas-api-tls" pod "c1617725-7d72-4a55-9084-e541e08ac95e" (UID: "c1617725-7d72-4a55-9084-e541e08ac95e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:36:27.010539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.010487 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1617725-7d72-4a55-9084-e541e08ac95e-kube-api-access-dmm2t" pod "c1617725-7d72-4a55-9084-e541e08ac95e" (UID: "c1617725-7d72-4a55-9084-e541e08ac95e"). InnerVolumeSpecName "kube-api-access-dmm2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:27.015715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.015680 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:27.089420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.067845 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmm2t\" (UniqueName: \"kubernetes.io/projected/c1617725-7d72-4a55-9084-e541e08ac95e-kube-api-access-dmm2t\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:27.089420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.067888 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:27.089420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.067904 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1617725-7d72-4a55-9084-e541e08ac95e-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:27.089420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.067920 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1617725-7d72-4a55-9084-e541e08ac95e-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:27.159041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.158950 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:27.347429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.344917 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:27.642418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.635769 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" event={"ID":"c1617725-7d72-4a55-9084-e541e08ac95e","Type":"ContainerDied","Data":"878a5f45937a56f2aa6eae489c3c73355be1462c9d318ba0debd3ef7cfac6d72"} Oct 29 22:36:27.642418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.635864 5610 scope.go:122] "RemoveContainer" containerID="e8f324ff6bd2c3be61141542ce0725255233327bbda504aa47dae088d2139a94" Oct 29 22:36:27.722416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.715969 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg" Oct 29 22:36:27.889420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.888755 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg"] Oct 29 22:36:27.898217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:27.898190 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1d9591d-5585d69dc8-p79pg"] Oct 29 22:36:28.984418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:28.983112 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:29.639002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:29.638956 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1617725-7d72-4a55-9084-e541e08ac95e" path="/var/lib/kubelet/pods/c1617725-7d72-4a55-9084-e541e08ac95e/volumes" Oct 29 22:36:29.808000 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:29.807951 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" podUID="93721f18-92c5-42ff-aca9-29b79893e6d3" containerName="payload-pre-processing" containerID="cri-o://87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406" gracePeriod=25 Oct 29 22:36:30.208016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.207991 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:36:30.246057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.246021 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93721f18-92c5-42ff-aca9-29b79893e6d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93721f18-92c5-42ff-aca9-29b79893e6d3-plugins-config-volume\") pod \"93721f18-92c5-42ff-aca9-29b79893e6d3\" (UID: \"93721f18-92c5-42ff-aca9-29b79893e6d3\") " Oct 29 22:36:30.246057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.246063 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93721f18-92c5-42ff-aca9-29b79893e6d3-kube-api-access-25cph\" (UniqueName: \"kubernetes.io/projected/93721f18-92c5-42ff-aca9-29b79893e6d3-kube-api-access-25cph\") pod \"93721f18-92c5-42ff-aca9-29b79893e6d3\" (UID: \"93721f18-92c5-42ff-aca9-29b79893e6d3\") " Oct 29 22:36:30.246493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.246468 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93721f18-92c5-42ff-aca9-29b79893e6d3-plugins-config-volume" pod "93721f18-92c5-42ff-aca9-29b79893e6d3" (UID: "93721f18-92c5-42ff-aca9-29b79893e6d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:30.247283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.247255 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93721f18-92c5-42ff-aca9-29b79893e6d3-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:30.250136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.250111 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93721f18-92c5-42ff-aca9-29b79893e6d3-kube-api-access-25cph" pod "93721f18-92c5-42ff-aca9-29b79893e6d3" (UID: "93721f18-92c5-42ff-aca9-29b79893e6d3"). InnerVolumeSpecName "kube-api-access-25cph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:30.348437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.348388 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25cph\" (UniqueName: \"kubernetes.io/projected/93721f18-92c5-42ff-aca9-29b79893e6d3-kube-api-access-25cph\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:30.550072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.549985 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:30.679424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.679111 5610 generic.go:356] "Generic (PLEG): container finished" podID="93721f18-92c5-42ff-aca9-29b79893e6d3" containerID="87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406" exitCode=0 Oct 29 22:36:30.679626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.679474 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" event={"ID":"93721f18-92c5-42ff-aca9-29b79893e6d3","Type":"ContainerDied","Data":"87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406"} Oct 29 22:36:30.679626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.679556 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" event={"ID":"93721f18-92c5-42ff-aca9-29b79893e6d3","Type":"ContainerDied","Data":"7f3b32db758e4dcf98895ef7fccfe85112b169755be11288acc7a5bab22c0338"} Oct 29 22:36:30.679626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.679607 5610 scope.go:122] "RemoveContainer" containerID="87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406" Oct 29 22:36:30.679829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.679811 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt" Oct 29 22:36:30.693083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.693053 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:30.702742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.702720 5610 scope.go:122] "RemoveContainer" containerID="87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406" Oct 29 22:36:30.703056 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:36:30.703034 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406\": container with ID starting with 87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406 not found: ID does not exist" containerID="87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406" Oct 29 22:36:30.703132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.703069 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406"} err="failed to get container status \"87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406\": rpc error: code = NotFound desc = could not find container \"87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406\": container with ID starting with 87a68e84f7156942613548f0dbf74bdfbea95705e9557beaafe5bd7a12bc9406 not found: ID does not exist" Oct 29 22:36:30.748039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.747995 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt"] Oct 29 22:36:30.786010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.785974 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e1d9591d-58975cb8ff-pssgt"] Oct 29 22:36:30.935484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:30.934031 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:31.638142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:31.638109 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93721f18-92c5-42ff-aca9-29b79893e6d3" path="/var/lib/kubelet/pods/93721f18-92c5-42ff-aca9-29b79893e6d3/volumes" Oct 29 22:36:32.237078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.237036 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:32.395805 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.395772 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:32.576091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.576007 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl"] Oct 29 22:36:32.638486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.614529 5610 cpu_manager.go:414] "containerMap: removing container" podUID="93721f18-92c5-42ff-aca9-29b79893e6d3" containerName="payload-pre-processing" Oct 29 22:36:32.654507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.654458 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93721f18-92c5-42ff-aca9-29b79893e6d3" containerName="payload-pre-processing" Oct 29 22:36:32.654732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.654718 5610 cpu_manager.go:414] "containerMap: removing container" podUID="c1617725-7d72-4a55-9084-e541e08ac95e" containerName="maas-api" Oct 29 22:36:32.654865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.654848 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1617725-7d72-4a55-9084-e541e08ac95e" containerName="maas-api" Oct 29 22:36:32.667713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.667673 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="93721f18-92c5-42ff-aca9-29b79893e6d3" containerName="payload-pre-processing" Oct 29 22:36:32.678513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.678482 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1617725-7d72-4a55-9084-e541e08ac95e" containerName="maas-api" Oct 29 22:36:32.859701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.859614 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:32.892801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.888432 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-afdbe8\"" Oct 29 22:36:32.892801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.889330 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-afdbe8-dockercfg-ws9h7\"" Oct 29 22:36:32.892801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:32.891748 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl"] Oct 29 22:36:33.018498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.018413 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36ddb4-7658-4597-a881-a9c36e7fb289-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl\" (UID: \"bd36ddb4-7658-4597-a881-a9c36e7fb289\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:33.018773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.018750 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwt2\" (UniqueName: \"kubernetes.io/projected/bd36ddb4-7658-4597-a881-a9c36e7fb289-kube-api-access-gdwt2\") pod \"payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl\" (UID: \"bd36ddb4-7658-4597-a881-a9c36e7fb289\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:33.120796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.120714 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36ddb4-7658-4597-a881-a9c36e7fb289-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl\" (UID: \"bd36ddb4-7658-4597-a881-a9c36e7fb289\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:33.120796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.120787 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwt2\" (UniqueName: \"kubernetes.io/projected/bd36ddb4-7658-4597-a881-a9c36e7fb289-kube-api-access-gdwt2\") pod \"payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl\" (UID: \"bd36ddb4-7658-4597-a881-a9c36e7fb289\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:33.121443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.121379 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36ddb4-7658-4597-a881-a9c36e7fb289-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl\" (UID: \"bd36ddb4-7658-4597-a881-a9c36e7fb289\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:33.141060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.141027 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwt2\" (UniqueName: \"kubernetes.io/projected/bd36ddb4-7658-4597-a881-a9c36e7fb289-kube-api-access-gdwt2\") pod \"payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl\" (UID: \"bd36ddb4-7658-4597-a881-a9c36e7fb289\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:33.288596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.288560 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:33.327361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.327332 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:33.825517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.825491 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl"] Oct 29 22:36:33.827640 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:36:33.827609 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd36ddb4_7658_4597_a881_a9c36e7fb289.slice/crio-f15ee260857a7b545b21954710d70a37cdfe078e595a4837055e05e4f1d07b3d WatchSource:0}: Error finding container f15ee260857a7b545b21954710d70a37cdfe078e595a4837055e05e4f1d07b3d: Status 404 returned error can't find the container with id f15ee260857a7b545b21954710d70a37cdfe078e595a4837055e05e4f1d07b3d Oct 29 22:36:33.862205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:33.862177 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" event={"ID":"bd36ddb4-7658-4597-a881-a9c36e7fb289","Type":"ContainerStarted","Data":"f15ee260857a7b545b21954710d70a37cdfe078e595a4837055e05e4f1d07b3d"} Oct 29 22:36:34.865947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:34.864127 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:34.898091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:34.896695 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" event={"ID":"bd36ddb4-7658-4597-a881-a9c36e7fb289","Type":"ContainerStarted","Data":"14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5"} Oct 29 22:36:34.898091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:34.898020 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:37.046762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.046722 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:37.296572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.296531 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:37.922872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.922779 5610 generic.go:356] "Generic (PLEG): container finished" podID="0ebc2f04-1165-4173-9084-8fe169540527" containerID="d0b0880c42e09b0787d38fcdb290dc21d01fc3dc05699be3f48935d01521b354" exitCode=0 Oct 29 22:36:37.922872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.922848 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" event={"ID":"0ebc2f04-1165-4173-9084-8fe169540527","Type":"ContainerDied","Data":"d0b0880c42e09b0787d38fcdb290dc21d01fc3dc05699be3f48935d01521b354"} Oct 29 22:36:37.923333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.923316 5610 scope.go:122] "RemoveContainer" containerID="d0b0880c42e09b0787d38fcdb290dc21d01fc3dc05699be3f48935d01521b354" Oct 29 22:36:37.924971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.924940 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:36:37.925059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.925011 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:36:37.925059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.925042 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:36:37.978055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:37.978000 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" podStartSLOduration=5.977982936 podStartE2EDuration="5.977982936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:34.965314047 +0000 UTC m=+1257.446631542" watchObservedRunningTime="2026-10-29 22:36:37.977982936 +0000 UTC m=+1260.459300429" Oct 29 22:36:38.868725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:38.868683 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:38.984694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:38.984661 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" event={"ID":"0ebc2f04-1165-4173-9084-8fe169540527","Type":"ContainerStarted","Data":"8692cde8c879abb120f3d0efb4f9e17195df3f775f8a58f9211a9635e36eddd8"} Oct 29 22:36:39.117423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.116673 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:39.517701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.517668 5610 scope.go:122] "RemoveContainer" containerID="64692eaa642e17626b136a11c78b300db19070c93f7773d025945e06dc52fc32" Oct 29 22:36:39.584116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.584068 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:39.652867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.652828 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:36:39.653926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.653901 5610 patch_prober.go:29] interesting pod/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:36:39.654036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.653951 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:36:39.942771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.942740 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5"] Oct 29 22:36:39.948356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.948330 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:39.964345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:39.963706 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8f1627-openshift-default-dockercfg-qjqk5\"" Oct 29 22:36:40.018650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.018620 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5"] Oct 29 22:36:40.029312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.029279 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.029472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.029347 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce818887-c1a6-452b-9584-432c17e64752-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.029519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.029486 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.029587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.029563 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.029722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.029664 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.029774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.029733 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.029890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.029870 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce818887-c1a6-452b-9584-432c17e64752-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.029926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.029916 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpmd\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-kube-api-access-9lpmd\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.030056 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.030043 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.055383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.055349 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.141795 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.141883 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.141979 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce818887-c1a6-452b-9584-432c17e64752-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.142007 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.142062 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.142190 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.142223 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.142386 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce818887-c1a6-452b-9584-432c17e64752-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.143485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.142466 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpmd\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-kube-api-access-9lpmd\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.144177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.143606 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-data\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.170419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.158181 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-credential-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.184434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.180716 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-certs\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.184434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.181627 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce818887-c1a6-452b-9584-432c17e64752-istiod-ca-cert\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.184434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.183192 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce818887-c1a6-452b-9584-432c17e64752-istio-podinfo\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.188524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.185034 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-envoy\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.194470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.192195 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-socket\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.234530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.234492 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:40.237427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.235292 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpmd\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-kube-api-access-9lpmd\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.245980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.245946 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-istio-token\") pod \"e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.317418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.310702 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:40.657152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.657119 5610 patch_prober.go:29] interesting pod/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:36:40.657460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:40.657384 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:36:41.103470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:41.102007 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:41.146423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:41.140111 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5"] Oct 29 22:36:41.268418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:41.253472 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:36:41.268418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:41.253597 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:36:41.268418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:41.253649 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:36:41.654562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:41.654459 5610 patch_prober.go:29] interesting pod/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:36:41.654959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:41.654897 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:36:41.936804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:41.936714 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:36:42.033036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:42.032995 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" event={"ID":"ce818887-c1a6-452b-9584-432c17e64752","Type":"ContainerStarted","Data":"f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0"} Oct 29 22:36:42.033250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:42.033066 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" event={"ID":"ce818887-c1a6-452b-9584-432c17e64752","Type":"ContainerStarted","Data":"eaff0b3c791184356053c3079677a909463b23071d32389577f066f93851cd3f"} Oct 29 22:36:42.101437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:42.101362 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" podStartSLOduration=3.101347258 podStartE2EDuration="3.101347258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:36:42.085882992 +0000 UTC m=+1264.567200486" watchObservedRunningTime="2026-10-29 22:36:42.101347258 +0000 UTC m=+1264.582664751" Oct 29 22:36:42.319475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:42.317012 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:42.361804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:42.361765 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:42.653560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:42.653478 5610 patch_prober.go:29] interesting pod/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:36:42.653560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:42.653539 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:36:42.865590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:42.860141 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:43.107417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:43.107090 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:43.108197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:43.107950 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:36:43.657648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:43.657564 5610 patch_prober.go:29] interesting pod/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:36:43.657648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:43.657626 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:36:43.832418 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:36:43.831096 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:36:44.606424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:44.606261 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:44.660549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:44.654876 5610 patch_prober.go:29] interesting pod/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 29 22:36:44.660549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:44.654957 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 29 22:36:44.789420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:44.786480 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:44.943160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:44.943129 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:45.804245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:45.803902 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:46.076874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:46.076781 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:36:46.093314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:46.093282 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:36:46.735432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:46.730710 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:46.787376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:46.787324 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:36:46.926418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:46.913222 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:36:46.926418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:46.913566 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:36:47.046516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:47.038592 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 22:36:47.183447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:47.183325 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:47.612496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:47.612455 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:36:47.832430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:47.832239 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:36:47.981733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:47.981603 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:36:48.389221 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:48.389124 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:36:49.040416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:48.950670 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:36:49.040416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:48.950755 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:36:49.040416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:48.749512 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:36:49.441470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:49.393181 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:36:49.579564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:49.579526 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:36:49.760678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:49.760634 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:36:50.002745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:50.002518 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:36:50.021201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:50.021110 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:36:50.041409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:50.027056 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:50.181639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:50.181607 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:36:50.423580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:50.423498 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:36:50.453165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:50.453135 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:50.794421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:50.793763 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:52.100669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:52.100619 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw"] Oct 29 22:36:52.103242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:52.103204 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" containerID="cri-o://8692cde8c879abb120f3d0efb4f9e17195df3f775f8a58f9211a9635e36eddd8" gracePeriod=30 Oct 29 22:36:52.193970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:52.193927 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:53.001636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:53.001594 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:53.457248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:53.457214 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:53.638419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:53.637464 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:55.808533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:55.808493 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:55.947831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:55.947794 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:56.113031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:56.112887 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:57.026800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:57.026768 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:57.208424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:57.208248 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:57.366752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:57.366661 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:57.768088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:57.768035 5610 generic.go:356] "Generic (PLEG): container finished" podID="0ebc2f04-1165-4173-9084-8fe169540527" containerID="8692cde8c879abb120f3d0efb4f9e17195df3f775f8a58f9211a9635e36eddd8" exitCode=0 Oct 29 22:36:57.768413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:57.768358 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" event={"ID":"0ebc2f04-1165-4173-9084-8fe169540527","Type":"ContainerDied","Data":"8692cde8c879abb120f3d0efb4f9e17195df3f775f8a58f9211a9635e36eddd8"} Oct 29 22:36:57.768682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:57.768654 5610 scope.go:122] "RemoveContainer" containerID="d0b0880c42e09b0787d38fcdb290dc21d01fc3dc05699be3f48935d01521b354" Oct 29 22:36:58.164379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.164338 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:36:58.233560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.233519 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-data\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.233745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.233623 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-envoy\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.233745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.233663 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ebc2f04-1165-4173-9084-8fe169540527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc2f04-1165-4173-9084-8fe169540527-istio-podinfo\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.233745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.233735 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebc2f04-1165-4173-9084-8fe169540527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc2f04-1165-4173-9084-8fe169540527-istiod-ca-cert\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.233945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.233819 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-credential-socket\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.234032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.234012 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-certs\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.234097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.234078 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-kube-api-access-56vlf\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-kube-api-access-56vlf\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.234224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.234207 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-istio-token\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.234295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.234272 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-socket\") pod \"0ebc2f04-1165-4173-9084-8fe169540527\" (UID: \"0ebc2f04-1165-4173-9084-8fe169540527\") " Oct 29 22:36:58.234709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.234676 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-credential-socket" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:58.234984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.234959 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-certs" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:58.235220 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.235196 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-socket" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:58.235302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.235232 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.235302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.235256 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.235467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.235446 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebc2f04-1165-4173-9084-8fe169540527-istiod-ca-cert" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:36:58.235553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.233834 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-data" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:58.240566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.240535 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ebc2f04-1165-4173-9084-8fe169540527-istio-podinfo" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:36:58.240660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.240570 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-istio-token" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:58.240660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.240615 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-kube-api-access-56vlf" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "kube-api-access-56vlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:36:58.257798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.257735 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-envoy" pod "0ebc2f04-1165-4173-9084-8fe169540527" (UID: "0ebc2f04-1165-4173-9084-8fe169540527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:36:58.336656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.336623 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ebc2f04-1165-4173-9084-8fe169540527-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.336656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.336651 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vlf\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-kube-api-access-56vlf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.336656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.336663 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ebc2f04-1165-4173-9084-8fe169540527-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.336905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.336672 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.336905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.336680 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.336905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.336688 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ebc2f04-1165-4173-9084-8fe169540527-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.336905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.336695 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ebc2f04-1165-4173-9084-8fe169540527-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:36:58.352068 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.351993 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:58.778345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.778314 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" Oct 29 22:36:58.778345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.778333 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw" event={"ID":"0ebc2f04-1165-4173-9084-8fe169540527","Type":"ContainerDied","Data":"6f9a07323c0884baa0324391de40da89ac0e468594160e917afe8948a33a0c46"} Oct 29 22:36:58.778637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.778410 5610 scope.go:122] "RemoveContainer" containerID="8692cde8c879abb120f3d0efb4f9e17195df3f775f8a58f9211a9635e36eddd8" Oct 29 22:36:58.846708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.846673 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw"] Oct 29 22:36:58.855574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.855543 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e1d9591d-openshift-default-7c64b644c7-kstxw"] Oct 29 22:36:58.980082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.980028 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:36:58.980279 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:58.980109 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:36:59.078529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.077063 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:59.141105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.141074 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m"] Oct 29 22:36:59.141956 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.141934 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" Oct 29 22:36:59.142053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.141958 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" Oct 29 22:36:59.142053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.142014 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" Oct 29 22:36:59.142053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.142022 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" Oct 29 22:36:59.142185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.142174 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" Oct 29 22:36:59.146903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.146881 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:36:59.171735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.164599 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-e81b56\"" Oct 29 22:36:59.171735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.167360 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m"] Oct 29 22:36:59.219066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.219032 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6"] Oct 29 22:36:59.220258 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.220235 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebc2f04-1165-4173-9084-8fe169540527" containerName="istio-proxy" Oct 29 22:36:59.224295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.224260 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:36:59.225086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.225060 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:36:59.240572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.240535 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6"] Oct 29 22:36:59.241456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.241430 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-e81b56-dockercfg-6xjbq\"" Oct 29 22:36:59.296854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.296805 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpf6h\" (UniqueName: \"kubernetes.io/projected/3a59f3f1-c569-4d63-9896-42d459c6f7bd-kube-api-access-wpf6h\") pod \"payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6\" (UID: \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:36:59.297053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.296927 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fb649c-6c0b-456c-ae9b-c3e52195fc53-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m\" (UID: \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:36:59.297131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.297116 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxv42\" (UniqueName: \"kubernetes.io/projected/82fb649c-6c0b-456c-ae9b-c3e52195fc53-kube-api-access-rxv42\") pod \"payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m\" (UID: \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:36:59.297191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.297147 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a59f3f1-c569-4d63-9896-42d459c6f7bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6\" (UID: \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:36:59.461955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.461916 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpf6h\" (UniqueName: \"kubernetes.io/projected/3a59f3f1-c569-4d63-9896-42d459c6f7bd-kube-api-access-wpf6h\") pod \"payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6\" (UID: \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:36:59.462261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.462237 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fb649c-6c0b-456c-ae9b-c3e52195fc53-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m\" (UID: \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:36:59.462495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.462475 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxv42\" (UniqueName: \"kubernetes.io/projected/82fb649c-6c0b-456c-ae9b-c3e52195fc53-kube-api-access-rxv42\") pod \"payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m\" (UID: \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:36:59.462630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.462610 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a59f3f1-c569-4d63-9896-42d459c6f7bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6\" (UID: \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:36:59.463920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.463893 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a59f3f1-c569-4d63-9896-42d459c6f7bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6\" (UID: \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:36:59.465358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.465231 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fb649c-6c0b-456c-ae9b-c3e52195fc53-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m\" (UID: \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:36:59.751465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.745001 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxv42\" (UniqueName: \"kubernetes.io/projected/82fb649c-6c0b-456c-ae9b-c3e52195fc53-kube-api-access-rxv42\") pod \"payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m\" (UID: \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:36:59.751465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.748410 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpf6h\" (UniqueName: \"kubernetes.io/projected/3a59f3f1-c569-4d63-9896-42d459c6f7bd-kube-api-access-wpf6h\") pod \"payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6\" (UID: \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:36:59.751465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.749547 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:36:59.765459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.765205 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:36:59.857417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:36:59.857127 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebc2f04-1165-4173-9084-8fe169540527" path="/var/lib/kubelet/pods/0ebc2f04-1165-4173-9084-8fe169540527/volumes" Oct 29 22:37:00.193114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.193086 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6"] Oct 29 22:37:00.193918 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:00.193868 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a59f3f1_c569_4d63_9896_42d459c6f7bd.slice/crio-4d791f2fbb3ef559841f59a7efc6544d1a00b36b67c0cad35d8fd3bda3abe22f WatchSource:0}: Error finding container 4d791f2fbb3ef559841f59a7efc6544d1a00b36b67c0cad35d8fd3bda3abe22f: Status 404 returned error can't find the container with id 4d791f2fbb3ef559841f59a7efc6544d1a00b36b67c0cad35d8fd3bda3abe22f Oct 29 22:37:00.204386 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:00.204358 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fb649c_6c0b_456c_ae9b_c3e52195fc53.slice/crio-1ae4f1de37201a242f2534eaef17ffdfcac79792ab07eec1497dc478ddcc8bfb WatchSource:0}: Error finding container 1ae4f1de37201a242f2534eaef17ffdfcac79792ab07eec1497dc478ddcc8bfb: Status 404 returned error can't find the container with id 1ae4f1de37201a242f2534eaef17ffdfcac79792ab07eec1497dc478ddcc8bfb Oct 29 22:37:00.214747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.214723 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m"] Oct 29 22:37:00.462418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.460927 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:00.881032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.880977 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" event={"ID":"82fb649c-6c0b-456c-ae9b-c3e52195fc53","Type":"ContainerStarted","Data":"c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45"} Oct 29 22:37:00.881032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.881039 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" event={"ID":"82fb649c-6c0b-456c-ae9b-c3e52195fc53","Type":"ContainerStarted","Data":"1ae4f1de37201a242f2534eaef17ffdfcac79792ab07eec1497dc478ddcc8bfb"} Oct 29 22:37:00.881301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.881193 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:37:00.882837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.882812 5610 generic.go:356] "Generic (PLEG): container finished" podID="55441a0e-ab81-44df-8301-956c67fecc3b" containerID="1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57" exitCode=0 Oct 29 22:37:00.882981 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.882891 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" event={"ID":"55441a0e-ab81-44df-8301-956c67fecc3b","Type":"ContainerDied","Data":"1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57"} Oct 29 22:37:00.883460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.883437 5610 scope.go:122] "RemoveContainer" containerID="1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57" Oct 29 22:37:00.884939 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.884906 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" event={"ID":"3a59f3f1-c569-4d63-9896-42d459c6f7bd","Type":"ContainerStarted","Data":"83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976"} Oct 29 22:37:00.885044 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.884958 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" event={"ID":"3a59f3f1-c569-4d63-9896-42d459c6f7bd","Type":"ContainerStarted","Data":"4d791f2fbb3ef559841f59a7efc6544d1a00b36b67c0cad35d8fd3bda3abe22f"} Oct 29 22:37:00.885044 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.884989 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:37:00.885327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.885299 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:37:00.885386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.885372 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:37:00.885470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.885447 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:37:00.920619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:00.920572 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" podStartSLOduration=1.920554467 podStartE2EDuration="1.920554467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:00.919074516 +0000 UTC m=+1283.400392010" watchObservedRunningTime="2026-10-29 22:37:00.920554467 +0000 UTC m=+1283.401871960" Oct 29 22:37:01.010197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:01.010035 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" podStartSLOduration=2.009950191 podStartE2EDuration="2.009950191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:01.008898352 +0000 UTC m=+1283.490215849" watchObservedRunningTime="2026-10-29 22:37:01.009950191 +0000 UTC m=+1283.491267685" Oct 29 22:37:01.385164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:01.374652 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:01.525418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:01.524896 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:02.001723 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:02.001685 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" event={"ID":"55441a0e-ab81-44df-8301-956c67fecc3b","Type":"ContainerStarted","Data":"0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c"} Oct 29 22:37:02.105429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:02.105377 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:37:02.105800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:02.105778 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:37:02.116593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:02.116552 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 29 22:37:02.739372 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:02.739331 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:03.082957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:03.082854 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:37:03.083232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:03.083202 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 29 22:37:04.067861 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:04.067828 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:37:04.068473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:04.068437 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 29 22:37:04.400475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:04.399579 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:05.079414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:05.078495 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:37:05.079414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:05.078576 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 29 22:37:05.573417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:05.571115 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk"] Oct 29 22:37:05.573417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:05.571367 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" podUID="5cb9529e-6f36-46e2-ac66-3168ef39d7bd" containerName="maas-api" containerID="cri-o://9966deeb3f4617d72900d64e7bb06bda0bf54a2db55e8a67cc1a0f6c2a6644f6" gracePeriod=30 Oct 29 22:37:05.703495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:05.699091 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:06.085465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.084437 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:37:06.085465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.084501 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 29 22:37:06.501518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.499670 5610 generic.go:356] "Generic (PLEG): container finished" podID="5cb9529e-6f36-46e2-ac66-3168ef39d7bd" containerID="9966deeb3f4617d72900d64e7bb06bda0bf54a2db55e8a67cc1a0f6c2a6644f6" exitCode=0 Oct 29 22:37:06.501518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.499776 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" event={"ID":"5cb9529e-6f36-46e2-ac66-3168ef39d7bd","Type":"ContainerDied","Data":"9966deeb3f4617d72900d64e7bb06bda0bf54a2db55e8a67cc1a0f6c2a6644f6"} Oct 29 22:37:06.504343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.503991 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:37:06.678854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.677372 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-openshift-service-ca\") pod \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " Oct 29 22:37:06.678854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.677491 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-kube-api-access-bpq64\" (UniqueName: \"kubernetes.io/projected/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-kube-api-access-bpq64\") pod \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " Oct 29 22:37:06.678854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.677540 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls\") pod \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " Oct 29 22:37:06.678854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.677787 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-metrics-certs\") pod \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\" (UID: \"5cb9529e-6f36-46e2-ac66-3168ef39d7bd\") " Oct 29 22:37:06.731111 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.729793 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:06.731872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.731790 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-openshift-service-ca" pod "5cb9529e-6f36-46e2-ac66-3168ef39d7bd" (UID: "5cb9529e-6f36-46e2-ac66-3168ef39d7bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:37:06.781268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.781230 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:37:06.884497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.883602 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-metrics-certs" pod "5cb9529e-6f36-46e2-ac66-3168ef39d7bd" (UID: "5cb9529e-6f36-46e2-ac66-3168ef39d7bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:06.921765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.921706 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-kube-api-access-bpq64" pod "5cb9529e-6f36-46e2-ac66-3168ef39d7bd" (UID: "5cb9529e-6f36-46e2-ac66-3168ef39d7bd"). InnerVolumeSpecName "kube-api-access-bpq64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:37:06.924724 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.924579 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls" pod "5cb9529e-6f36-46e2-ac66-3168ef39d7bd" (UID: "5cb9529e-6f36-46e2-ac66-3168ef39d7bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:37:06.927552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.927484 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:37:06.927552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.927515 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpq64\" (UniqueName: \"kubernetes.io/projected/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-kube-api-access-bpq64\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:37:06.927552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.927528 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cb9529e-6f36-46e2-ac66-3168ef39d7bd-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:37:06.990608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:06.989445 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:07.079339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.070113 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:37:07.079339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.070174 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 29 22:37:07.274978 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.274815 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:07.533603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.533514 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" Oct 29 22:37:07.533603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.533553 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk" event={"ID":"5cb9529e-6f36-46e2-ac66-3168ef39d7bd","Type":"ContainerDied","Data":"0795956eaf74bad12f9ca20fb9fade77ca1e32855bede31ccc6731482ab1ec6f"} Oct 29 22:37:07.533781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.533626 5610 scope.go:122] "RemoveContainer" containerID="9966deeb3f4617d72900d64e7bb06bda0bf54a2db55e8a67cc1a0f6c2a6644f6" Oct 29 22:37:07.537527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.537502 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:07.595127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.595093 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk"] Oct 29 22:37:07.605554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.605524 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f922c6e9-6c5f4dd795-vwvbk"] Oct 29 22:37:07.638122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:07.638091 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb9529e-6f36-46e2-ac66-3168ef39d7bd" path="/var/lib/kubelet/pods/5cb9529e-6f36-46e2-ac66-3168ef39d7bd/volumes" Oct 29 22:37:08.084416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:08.083806 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 29 22:37:08.084416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:08.083880 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 29 22:37:09.028217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:09.028157 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:09.115422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:09.113817 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:37:09.115649 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:09.115501 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:37:09.120074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:09.120044 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 22:37:10.567021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.566982 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs"] Oct 29 22:37:10.567847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.567829 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5cb9529e-6f36-46e2-ac66-3168ef39d7bd" containerName="maas-api" Oct 29 22:37:10.567904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.567848 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb9529e-6f36-46e2-ac66-3168ef39d7bd" containerName="maas-api" Oct 29 22:37:10.568024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.568015 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb9529e-6f36-46e2-ac66-3168ef39d7bd" containerName="maas-api" Oct 29 22:37:10.571718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.571693 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.591980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.591946 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs"] Oct 29 22:37:10.592718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.592694 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8f1627-serving-cert\"" Oct 29 22:37:10.688089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.688058 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:10.741380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.741345 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.741567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.741551 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbt2\" (UniqueName: \"kubernetes.io/projected/f7d38d15-b38f-423b-aeef-68f36963738c-kube-api-access-8sbt2\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.741616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.741575 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.741740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.741728 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d38d15-b38f-423b-aeef-68f36963738c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.832476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.827212 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:10.851417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.843809 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d38d15-b38f-423b-aeef-68f36963738c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.851417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.844066 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.851417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.844345 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbt2\" (UniqueName: \"kubernetes.io/projected/f7d38d15-b38f-423b-aeef-68f36963738c-kube-api-access-8sbt2\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.851417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.844381 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.851417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.848539 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d38d15-b38f-423b-aeef-68f36963738c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.879812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.879770 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.914193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.914095 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:10.922433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:10.922137 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbt2\" (UniqueName: \"kubernetes.io/projected/f7d38d15-b38f-423b-aeef-68f36963738c-kube-api-access-8sbt2\") pod \"maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:11.192229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:11.192139 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:11.537773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:11.537741 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs"] Oct 29 22:37:11.538036 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:11.538002 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d38d15_b38f_423b_aeef_68f36963738c.slice/crio-343fb2cba13a9433a3b57e74d1f5f9e3ac9e1d5b7ff5cd8e89ce2da635b9044f WatchSource:0}: Error finding container 343fb2cba13a9433a3b57e74d1f5f9e3ac9e1d5b7ff5cd8e89ce2da635b9044f: Status 404 returned error can't find the container with id 343fb2cba13a9433a3b57e74d1f5f9e3ac9e1d5b7ff5cd8e89ce2da635b9044f Oct 29 22:37:11.573703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:11.573670 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" event={"ID":"f7d38d15-b38f-423b-aeef-68f36963738c","Type":"ContainerStarted","Data":"343fb2cba13a9433a3b57e74d1f5f9e3ac9e1d5b7ff5cd8e89ce2da635b9044f"} Oct 29 22:37:12.020095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:12.020064 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 22:37:12.027609 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:12.027580 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 22:37:12.556238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:12.556098 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:12.606911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:12.604215 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" event={"ID":"f7d38d15-b38f-423b-aeef-68f36963738c","Type":"ContainerStarted","Data":"252cf9424398b9042c5340177e4929825ddb62cf40d105f547fdd3e2d31b6abf"} Oct 29 22:37:12.606911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:12.605440 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:12.657110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:12.656195 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" podStartSLOduration=2.222166999 podStartE2EDuration="2.656174584s" totalImagesPullingTime="434.007585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:12.654508301 +0000 UTC m=+1295.135825795" watchObservedRunningTime="2026-10-29 22:37:12.656174584 +0000 UTC m=+1295.137492080" Oct 29 22:37:13.714164 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:37:13.714119 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:37:13.874105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:13.874049 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:14.010856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:14.010775 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:14.233049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:14.233016 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:14.388869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:14.388769 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:14.738156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:14.738037 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:14.892701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:14.892663 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:15.040941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:15.040842 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:15.858127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:15.858089 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:17.346566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:17.346529 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:18.841422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:18.818469 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j"] Oct 29 22:37:18.968048 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:18.945285 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:18.987418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:18.985021 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.022979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.018122 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-e81b56-serving-cert\"" Oct 29 22:37:19.142337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.126906 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j"] Oct 29 22:37:19.142337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.141735 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:19.257027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.256865 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.257464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.257428 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.258012 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.257992 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rpj\" (UniqueName: \"kubernetes.io/projected/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-kube-api-access-p8rpj\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.258336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.258315 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.380416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.363075 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.380416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.369236 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.380416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.369305 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.380416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.369576 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rpj\" (UniqueName: \"kubernetes.io/projected/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-kube-api-access-p8rpj\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.380416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.376513 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.380416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.377193 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.404740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.404660 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.405217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.405192 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rpj\" (UniqueName: \"kubernetes.io/projected/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-kube-api-access-p8rpj\") pod \"maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.484567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.484529 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:19.653544 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.653516 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 22:37:19.799155 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:19.799051 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e8c33c_39e3_4aaa_a59a_4d78a7e6cc6a.slice/crio-147e78006a8499f415ef8a4e4be9eaa0dc784d0ed2abc63cae9ca30cfe73d85f WatchSource:0}: Error finding container 147e78006a8499f415ef8a4e4be9eaa0dc784d0ed2abc63cae9ca30cfe73d85f: Status 404 returned error can't find the container with id 147e78006a8499f415ef8a4e4be9eaa0dc784d0ed2abc63cae9ca30cfe73d85f Oct 29 22:37:19.802145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:19.802112 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j"] Oct 29 22:37:20.722454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:20.722370 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" event={"ID":"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a","Type":"ContainerStarted","Data":"6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f"} Oct 29 22:37:20.722933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:20.722503 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" event={"ID":"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a","Type":"ContainerStarted","Data":"147e78006a8499f415ef8a4e4be9eaa0dc784d0ed2abc63cae9ca30cfe73d85f"} Oct 29 22:37:20.722933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:20.722695 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:20.797331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:20.785342 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" podStartSLOduration=2.454118296 podStartE2EDuration="2.785321985s" totalImagesPullingTime="331.203689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:20.784627903 +0000 UTC m=+1303.265945397" watchObservedRunningTime="2026-10-29 22:37:20.785321985 +0000 UTC m=+1303.266639509" Oct 29 22:37:21.114080 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:21.113956 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:21.536439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:21.533131 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:21.970740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:21.970705 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:22.155643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:22.155606 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:22.305576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:22.305430 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:22.477660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:22.477622 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:22.678470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:22.678295 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:22.981218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:22.981001 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:23.155450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:23.148204 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:23.814588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:23.814515 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:23.976447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:23.969082 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:24.579416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:24.554084 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:24.770274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:24.770232 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:25.377230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:25.377188 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:26.604337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:26.604302 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:26.758644 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:26.758609 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:26.791376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:26.791346 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 22:37:27.326438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:27.326389 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:27.667507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:27.667409 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:27.820553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:27.820520 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:28.229762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:28.229725 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:28.403574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:28.403535 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:28.728434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:28.701320 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:28.728434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:28.727545 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z"] Oct 29 22:37:28.956482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:28.944117 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z"] Oct 29 22:37:28.956482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:28.945533 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:28.997668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:28.997581 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-84304f\"" Oct 29 22:37:29.017324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.016832 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-84304f-dockercfg-294vt\"" Oct 29 22:37:29.053842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.053802 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28642a96-d2b2-40e5-b32e-7fa96f97efe9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z\" (UID: \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:29.054065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.054019 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9vt8\" (UniqueName: \"kubernetes.io/projected/28642a96-d2b2-40e5-b32e-7fa96f97efe9-kube-api-access-v9vt8\") pod \"payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z\" (UID: \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:29.156161 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.156119 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vt8\" (UniqueName: \"kubernetes.io/projected/28642a96-d2b2-40e5-b32e-7fa96f97efe9-kube-api-access-v9vt8\") pod \"payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z\" (UID: \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:29.156343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.156329 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28642a96-d2b2-40e5-b32e-7fa96f97efe9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z\" (UID: \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:29.157062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.157034 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28642a96-d2b2-40e5-b32e-7fa96f97efe9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z\" (UID: \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:29.177849 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.177814 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9vt8\" (UniqueName: \"kubernetes.io/projected/28642a96-d2b2-40e5-b32e-7fa96f97efe9-kube-api-access-v9vt8\") pod \"payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z\" (UID: \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:29.340121 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.340043 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:29.654631 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:29.654601 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z"] Oct 29 22:37:29.654740 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:29.654711 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28642a96_d2b2_40e5_b32e_7fa96f97efe9.slice/crio-e27e84b22aca67b9a37d7b52e0f63a7fdf64bac2440d991942dc628a248ad8e9 WatchSource:0}: Error finding container e27e84b22aca67b9a37d7b52e0f63a7fdf64bac2440d991942dc628a248ad8e9: Status 404 returned error can't find the container with id e27e84b22aca67b9a37d7b52e0f63a7fdf64bac2440d991942dc628a248ad8e9 Oct 29 22:37:30.030162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.030098 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:30.197420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.195651 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:30.197420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.195794 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" event={"ID":"28642a96-d2b2-40e5-b32e-7fa96f97efe9","Type":"ContainerStarted","Data":"263cfeb7a14bd60d79aefbb844c06c99407788dbc87a39f41074abe7050f6d4a"} Oct 29 22:37:30.197420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.195852 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" event={"ID":"28642a96-d2b2-40e5-b32e-7fa96f97efe9","Type":"ContainerStarted","Data":"e27e84b22aca67b9a37d7b52e0f63a7fdf64bac2440d991942dc628a248ad8e9"} Oct 29 22:37:30.197420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.196125 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:30.246263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.246207 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" podStartSLOduration=2.246191301 podStartE2EDuration="2.246191301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:30.243525464 +0000 UTC m=+1312.724842957" watchObservedRunningTime="2026-10-29 22:37:30.246191301 +0000 UTC m=+1312.727508835" Oct 29 22:37:30.259689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.259657 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n"] Oct 29 22:37:30.272945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.272913 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:30.292762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.292723 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n"] Oct 29 22:37:30.296937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.296914 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d478b36e\"" Oct 29 22:37:30.298487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.297520 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d478b36e-dockercfg-4z7v2\"" Oct 29 22:37:30.447427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.444521 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxwz\" (UniqueName: \"kubernetes.io/projected/73580531-7db0-4a6a-9f0d-14ac98b530bd-kube-api-access-ssxwz\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n\" (UID: \"73580531-7db0-4a6a-9f0d-14ac98b530bd\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:30.447427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.444698 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73580531-7db0-4a6a-9f0d-14ac98b530bd-plugins-config-volume\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n\" (UID: \"73580531-7db0-4a6a-9f0d-14ac98b530bd\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:30.561143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.561049 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxwz\" (UniqueName: \"kubernetes.io/projected/73580531-7db0-4a6a-9f0d-14ac98b530bd-kube-api-access-ssxwz\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n\" (UID: \"73580531-7db0-4a6a-9f0d-14ac98b530bd\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:30.562964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.562932 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73580531-7db0-4a6a-9f0d-14ac98b530bd-plugins-config-volume\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n\" (UID: \"73580531-7db0-4a6a-9f0d-14ac98b530bd\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:30.564307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.564039 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73580531-7db0-4a6a-9f0d-14ac98b530bd-plugins-config-volume\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n\" (UID: \"73580531-7db0-4a6a-9f0d-14ac98b530bd\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:30.612910 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.612876 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxwz\" (UniqueName: \"kubernetes.io/projected/73580531-7db0-4a6a-9f0d-14ac98b530bd-kube-api-access-ssxwz\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n\" (UID: \"73580531-7db0-4a6a-9f0d-14ac98b530bd\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:30.630852 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:30.630812 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:31.172171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.172137 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n"] Oct 29 22:37:31.175753 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:31.175677 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73580531_7db0_4a6a_9f0d_14ac98b530bd.slice/crio-1ad739adca001d31326e1fd84dea74f15a923a503d20b9efc5c24c355151cdc1 WatchSource:0}: Error finding container 1ad739adca001d31326e1fd84dea74f15a923a503d20b9efc5c24c355151cdc1: Status 404 returned error can't find the container with id 1ad739adca001d31326e1fd84dea74f15a923a503d20b9efc5c24c355151cdc1 Oct 29 22:37:31.214059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.214020 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" event={"ID":"73580531-7db0-4a6a-9f0d-14ac98b530bd","Type":"ContainerStarted","Data":"1ad739adca001d31326e1fd84dea74f15a923a503d20b9efc5c24c355151cdc1"} Oct 29 22:37:31.554670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.554632 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:31.675413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.675365 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s"] Oct 29 22:37:31.681558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.681524 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:31.696245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.696210 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:31.701036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.700997 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s"] Oct 29 22:37:31.702930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.702903 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-c7b96a\"" Oct 29 22:37:31.703104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.703092 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-c7b96a-dockercfg-plnq8\"" Oct 29 22:37:31.735647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.735612 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e007c1-145b-4e23-92b4-2cf4933c3792-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s\" (UID: \"27e007c1-145b-4e23-92b4-2cf4933c3792\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:31.735833 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.735765 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79k7n\" (UniqueName: \"kubernetes.io/projected/27e007c1-145b-4e23-92b4-2cf4933c3792-kube-api-access-79k7n\") pod \"payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s\" (UID: \"27e007c1-145b-4e23-92b4-2cf4933c3792\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:31.877712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.876559 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e007c1-145b-4e23-92b4-2cf4933c3792-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s\" (UID: \"27e007c1-145b-4e23-92b4-2cf4933c3792\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:31.877712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.876854 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79k7n\" (UniqueName: \"kubernetes.io/projected/27e007c1-145b-4e23-92b4-2cf4933c3792-kube-api-access-79k7n\") pod \"payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s\" (UID: \"27e007c1-145b-4e23-92b4-2cf4933c3792\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:31.896415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.888336 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e007c1-145b-4e23-92b4-2cf4933c3792-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s\" (UID: \"27e007c1-145b-4e23-92b4-2cf4933c3792\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:31.933415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.925253 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc"] Oct 29 22:37:31.996753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:31.996711 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:32.039421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.032899 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc"] Oct 29 22:37:32.185482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.179823 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79k7n\" (UniqueName: \"kubernetes.io/projected/27e007c1-145b-4e23-92b4-2cf4933c3792-kube-api-access-79k7n\") pod \"payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s\" (UID: \"27e007c1-145b-4e23-92b4-2cf4933c3792\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:32.284783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.284737 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkfg\" (UniqueName: \"kubernetes.io/projected/61175f67-8f4d-42a3-91dc-fb62b86cdd46-kube-api-access-gtkfg\") pod \"payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc\" (UID: \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:32.289158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.289027 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61175f67-8f4d-42a3-91dc-fb62b86cdd46-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc\" (UID: \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:32.305354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.303888 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" event={"ID":"73580531-7db0-4a6a-9f0d-14ac98b530bd","Type":"ContainerStarted","Data":"6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51"} Oct 29 22:37:32.305354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.305294 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:32.316690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.316656 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:32.382036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.381983 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" podStartSLOduration=2.381963642 podStartE2EDuration="2.381963642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:32.369881148 +0000 UTC m=+1314.851198642" watchObservedRunningTime="2026-10-29 22:37:32.381963642 +0000 UTC m=+1314.863281146" Oct 29 22:37:32.393313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.392527 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkfg\" (UniqueName: \"kubernetes.io/projected/61175f67-8f4d-42a3-91dc-fb62b86cdd46-kube-api-access-gtkfg\") pod \"payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc\" (UID: \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:32.393313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.392615 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61175f67-8f4d-42a3-91dc-fb62b86cdd46-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc\" (UID: \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:32.395420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.395056 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61175f67-8f4d-42a3-91dc-fb62b86cdd46-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc\" (UID: \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:32.413475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.413443 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkfg\" (UniqueName: \"kubernetes.io/projected/61175f67-8f4d-42a3-91dc-fb62b86cdd46-kube-api-access-gtkfg\") pod \"payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc\" (UID: \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:32.506271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.505383 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:32.580417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.568888 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:32.757426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:32.750132 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s"] Oct 29 22:37:33.070005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.069962 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc"] Oct 29 22:37:33.333495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.333332 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" event={"ID":"27e007c1-145b-4e23-92b4-2cf4933c3792","Type":"ContainerStarted","Data":"3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7"} Oct 29 22:37:33.334053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.333569 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:33.334053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.333675 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" event={"ID":"27e007c1-145b-4e23-92b4-2cf4933c3792","Type":"ContainerStarted","Data":"cb37c7b376a398a6aab76fefb556c0af6eb0eb4fbb9fd4bbb49da7bc0a1e38b4"} Oct 29 22:37:33.337294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.337253 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" event={"ID":"61175f67-8f4d-42a3-91dc-fb62b86cdd46","Type":"ContainerStarted","Data":"4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba"} Oct 29 22:37:33.337446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.337296 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" event={"ID":"61175f67-8f4d-42a3-91dc-fb62b86cdd46","Type":"ContainerStarted","Data":"481f2a4ceec58ade39e0d448d31f9fbcbeecd664e072d39341d521e2f15d728a"} Oct 29 22:37:33.337446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.337322 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:33.373665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.373598 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" podStartSLOduration=2.373579724 podStartE2EDuration="2.373579724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:33.370376952 +0000 UTC m=+1315.851694436" watchObservedRunningTime="2026-10-29 22:37:33.373579724 +0000 UTC m=+1315.854897220" Oct 29 22:37:33.409695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:33.409634 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" podStartSLOduration=2.409617903 podStartE2EDuration="2.409617903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:33.406684656 +0000 UTC m=+1315.888002149" watchObservedRunningTime="2026-10-29 22:37:33.409617903 +0000 UTC m=+1315.890935396" Oct 29 22:37:34.530047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:34.530003 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:36.356942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:36.356889 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:37.404645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:37.404611 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:38.094430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:38.088952 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:39.794416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:39.790728 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:40.590104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:40.590055 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:41.217835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:41.217806 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 22:37:41.300089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:41.300051 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:42.451593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:42.451554 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:42.611576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:42.611544 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:42.949703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:42.949663 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:43.149414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:43.146948 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:43.321081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:43.320987 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:43.718141 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:37:43.718106 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:37:44.347825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:44.347794 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:37:44.348281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:44.348258 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:37:44.348449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:44.348330 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:37:44.918040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:44.910814 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:46.626357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:46.626286 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:47.960889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:47.960853 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:48.175739 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:48.175704 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:48.353722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:48.351889 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:48.518041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:48.517946 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:48.673641 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:48.672924 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:48.963429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:48.959661 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:49.169418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:49.165804 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:50.271273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:50.271225 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:50.429122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:50.429083 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:50.650470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:50.647155 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:50.795777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:50.795741 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:51.285381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:51.285338 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:51.541518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:51.539546 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:52.843141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:52.843102 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9"] Oct 29 22:37:52.849915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:52.849889 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:52.875620 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:52.875591 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d478b36e-serving-cert\"" Oct 29 22:37:52.880487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:52.880460 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9"] Oct 29 22:37:52.953050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:52.953007 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-metrics-certs\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:52.953245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:52.953129 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8644924f-1343-4fde-a1f5-e919ab41e380-openshift-service-ca\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:52.953245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:52.953180 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbwd\" (UniqueName: \"kubernetes.io/projected/8644924f-1343-4fde-a1f5-e919ab41e380-kube-api-access-xhbwd\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:52.953726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:52.953248 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.011361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.011331 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:53.054816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.054775 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-metrics-certs\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.054985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.054843 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8644924f-1343-4fde-a1f5-e919ab41e380-openshift-service-ca\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.054985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.054872 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbwd\" (UniqueName: \"kubernetes.io/projected/8644924f-1343-4fde-a1f5-e919ab41e380-kube-api-access-xhbwd\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.054985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.054935 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.055126 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:37:53.055102 5610 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-serving-cert: secret "maas-api-e2e-mt-d478b36e-serving-cert" not found Oct 29 22:37:53.055215 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:37:53.055196 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls podName:8644924f-1343-4fde-a1f5-e919ab41e380 nodeName:}" failed. No retries permitted until 2026-10-29 22:37:53.555173806 +0000 UTC m=+1336.036491293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls") pod "maas-api-e2e-mt-d478b36e-75bc765888-685v9" (UID: "8644924f-1343-4fde-a1f5-e919ab41e380") : secret "maas-api-e2e-mt-d478b36e-serving-cert" not found Oct 29 22:37:53.055613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.055593 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8644924f-1343-4fde-a1f5-e919ab41e380-openshift-service-ca\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.057492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.057465 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-metrics-certs\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.091064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.091031 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbwd\" (UniqueName: \"kubernetes.io/projected/8644924f-1343-4fde-a1f5-e919ab41e380-kube-api-access-xhbwd\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.277190 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.277143 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:53.459236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.459157 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:53.613303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.613203 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.634414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.617978 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-685v9\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.655487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.643371 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:53.877186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.869314 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:53.877186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:53.875472 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:54.300494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:54.298673 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9"] Oct 29 22:37:54.300897 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:54.300683 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8644924f_1343_4fde_a1f5_e919ab41e380.slice/crio-3982343a7ea016357d50c3e3b7ccccfeab8334732353be1ffab75e4917295561 WatchSource:0}: Error finding container 3982343a7ea016357d50c3e3b7ccccfeab8334732353be1ffab75e4917295561: Status 404 returned error can't find the container with id 3982343a7ea016357d50c3e3b7ccccfeab8334732353be1ffab75e4917295561 Oct 29 22:37:55.199116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.160742 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:55.276306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.251505 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw"] Oct 29 22:37:55.294709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.294677 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:55.392054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.392024 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0ab33e\"" Oct 29 22:37:55.469112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.469034 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0ab33e-dockercfg-sggz8\"" Oct 29 22:37:55.520084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.497298 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:55.520084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.516566 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw"] Oct 29 22:37:55.521937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.521903 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" event={"ID":"8644924f-1343-4fde-a1f5-e919ab41e380","Type":"ContainerStarted","Data":"3982343a7ea016357d50c3e3b7ccccfeab8334732353be1ffab75e4917295561"} Oct 29 22:37:55.597418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.578742 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fef595a-be02-49f5-9287-1f0f47359337-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw\" (UID: \"5fef595a-be02-49f5-9287-1f0f47359337\") " pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:55.597418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.578891 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84x9\" (UniqueName: \"kubernetes.io/projected/5fef595a-be02-49f5-9287-1f0f47359337-kube-api-access-x84x9\") pod \"payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw\" (UID: \"5fef595a-be02-49f5-9287-1f0f47359337\") " pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:55.682137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.682102 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fef595a-be02-49f5-9287-1f0f47359337-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw\" (UID: \"5fef595a-be02-49f5-9287-1f0f47359337\") " pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:55.687177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.683379 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84x9\" (UniqueName: \"kubernetes.io/projected/5fef595a-be02-49f5-9287-1f0f47359337-kube-api-access-x84x9\") pod \"payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw\" (UID: \"5fef595a-be02-49f5-9287-1f0f47359337\") " pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:55.687930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.683235 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fef595a-be02-49f5-9287-1f0f47359337-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw\" (UID: \"5fef595a-be02-49f5-9287-1f0f47359337\") " pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:55.712247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.712214 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84x9\" (UniqueName: \"kubernetes.io/projected/5fef595a-be02-49f5-9287-1f0f47359337-kube-api-access-x84x9\") pod \"payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw\" (UID: \"5fef595a-be02-49f5-9287-1f0f47359337\") " pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:55.935099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:55.935014 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:56.315464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:56.315316 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:56.353188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:56.353159 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw"] Oct 29 22:37:56.354524 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:56.354496 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fef595a_be02_49f5_9287_1f0f47359337.slice/crio-303c622ab94e3bcb76d8c2b523fd59f4927ef0465f592b62c487bd46e89c23dd WatchSource:0}: Error finding container 303c622ab94e3bcb76d8c2b523fd59f4927ef0465f592b62c487bd46e89c23dd: Status 404 returned error can't find the container with id 303c622ab94e3bcb76d8c2b523fd59f4927ef0465f592b62c487bd46e89c23dd Oct 29 22:37:56.483426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:56.481452 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:56.692804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:56.692764 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" event={"ID":"8644924f-1343-4fde-a1f5-e919ab41e380","Type":"ContainerStarted","Data":"281d975ad03562c6b0653f499e8a60a9a872d688f61853f626b03ad9179dc490"} Oct 29 22:37:56.715529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:56.698534 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:37:56.808433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:56.794113 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" event={"ID":"5fef595a-be02-49f5-9287-1f0f47359337","Type":"ContainerStarted","Data":"303c622ab94e3bcb76d8c2b523fd59f4927ef0465f592b62c487bd46e89c23dd"} Oct 29 22:37:57.805359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:57.805322 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" event={"ID":"5fef595a-be02-49f5-9287-1f0f47359337","Type":"ContainerStarted","Data":"59cd074a91c6c2f703e21ce9474446ba28171fbce461aae0d114680a349fd4cc"} Oct 29 22:37:57.805909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:57.805432 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:37:57.861420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:57.859243 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" podStartSLOduration=4.596666995 podStartE2EDuration="5.859220252s" totalImagesPullingTime="1.262553257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:56.868104637 +0000 UTC m=+1339.349422135" watchObservedRunningTime="2026-10-29 22:37:57.859220252 +0000 UTC m=+1340.340537748" Oct 29 22:37:57.872418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:57.871358 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" podStartSLOduration=3.871340229 podStartE2EDuration="3.871340229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:37:57.844914446 +0000 UTC m=+1340.326231940" watchObservedRunningTime="2026-10-29 22:37:57.871340229 +0000 UTC m=+1340.352657723" Oct 29 22:37:58.131945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:58.117056 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:58.356787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:58.338757 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:58.410562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:58.410475 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs"] Oct 29 22:37:58.740427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:58.730658 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:37:58.930709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:58.930673 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs"] Oct 29 22:37:58.984873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:58.984846 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-6bb06a-dockercfg-6fsn9\"" Oct 29 22:37:58.987303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:58.987276 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-6bb06a\"" Oct 29 22:37:59.064542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.064505 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc1edc1-003a-4e57-9ab6-8c789418cb15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs\" (UID: \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:37:59.064977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.064952 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4t2t\" (UniqueName: \"kubernetes.io/projected/cdc1edc1-003a-4e57-9ab6-8c789418cb15-kube-api-access-c4t2t\") pod \"payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs\" (UID: \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:37:59.183156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.183119 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc1edc1-003a-4e57-9ab6-8c789418cb15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs\" (UID: \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:37:59.183547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.183521 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4t2t\" (UniqueName: \"kubernetes.io/projected/cdc1edc1-003a-4e57-9ab6-8c789418cb15-kube-api-access-c4t2t\") pod \"payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs\" (UID: \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:37:59.190357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.190328 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:37:59.201426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.201361 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc1edc1-003a-4e57-9ab6-8c789418cb15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs\" (UID: \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:37:59.258435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.248205 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4t2t\" (UniqueName: \"kubernetes.io/projected/cdc1edc1-003a-4e57-9ab6-8c789418cb15-kube-api-access-c4t2t\") pod \"payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs\" (UID: \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:37:59.391901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.391789 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:37:59.810635 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:37:59.810582 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc1edc1_003a_4e57_9ab6_8c789418cb15.slice/crio-4192836be480f5649b0bd14d18803517d4be9f56c51c47ab8b427ae8e4495b61 WatchSource:0}: Error finding container 4192836be480f5649b0bd14d18803517d4be9f56c51c47ab8b427ae8e4495b61: Status 404 returned error can't find the container with id 4192836be480f5649b0bd14d18803517d4be9f56c51c47ab8b427ae8e4495b61 Oct 29 22:37:59.810762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:37:59.810635 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs"] Oct 29 22:38:00.090796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:00.090682 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" event={"ID":"cdc1edc1-003a-4e57-9ab6-8c789418cb15","Type":"ContainerStarted","Data":"4192836be480f5649b0bd14d18803517d4be9f56c51c47ab8b427ae8e4495b61"} Oct 29 22:38:00.597423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:00.592943 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:00.741015 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:00.740960 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:01.213465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:01.200605 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" event={"ID":"cdc1edc1-003a-4e57-9ab6-8c789418cb15","Type":"ContainerStarted","Data":"0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f"} Oct 29 22:38:01.224748 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:01.216131 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:38:01.348420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:01.341032 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" podStartSLOduration=3.34100624 podStartE2EDuration="3.34100624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:01.284005662 +0000 UTC m=+1343.765323157" watchObservedRunningTime="2026-10-29 22:38:01.34100624 +0000 UTC m=+1343.822323756" Oct 29 22:38:01.992834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:01.992795 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:03.540386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:03.540346 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:03.711827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:03.711787 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:04.045855 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:04.045828 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:38:05.377878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:05.377846 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:06.952731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:06.941741 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:07.310376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:07.310278 5610 generic.go:356] "Generic (PLEG): container finished" podID="ce818887-c1a6-452b-9584-432c17e64752" containerID="f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0" exitCode=0 Oct 29 22:38:07.310376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:07.310352 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" event={"ID":"ce818887-c1a6-452b-9584-432c17e64752","Type":"ContainerDied","Data":"f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0"} Oct 29 22:38:07.310899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:07.310883 5610 scope.go:122] "RemoveContainer" containerID="f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0" Oct 29 22:38:07.313500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:07.313472 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:38:07.313600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:07.313541 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:38:07.313600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:07.313580 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:38:08.320191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:08.320159 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" event={"ID":"ce818887-c1a6-452b-9584-432c17e64752","Type":"ContainerStarted","Data":"d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9"} Oct 29 22:38:08.573565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:08.572696 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:09.042532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:09.042502 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:38:09.313653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:09.313558 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:38:09.314826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:09.314803 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:38:09.314931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:09.314852 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:38:10.313777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:10.313695 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:38:10.313777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:10.313769 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:38:10.477999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:10.477959 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:11.313888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:11.313850 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:38:11.314315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:11.313919 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:38:12.158960 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:12.158915 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:12.331514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:12.320834 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:38:12.331514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:12.320900 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:38:13.107047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:13.107006 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:38:13.289895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:13.288593 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 22:38:13.314719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:13.314358 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 29 22:38:13.314719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:13.314438 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 29 22:38:13.720411 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:38:13.720361 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:38:13.882940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:13.882881 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:14.318666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:14.318635 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:38:14.319981 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:14.319956 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 22:38:15.135740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:15.135703 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:15.528509 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:15.528477 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:17.306411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:17.301612 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:19.349992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:19.349947 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:21.039504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:21.032862 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:21.548538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:21.548489 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:22.668262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:22.668224 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:23.427045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:23.427012 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:23.591417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:23.584996 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:24.309982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:24.309948 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:24.783756 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:24.783720 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:25.412476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:25.412440 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:25.997361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:25.997326 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:26.270034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:26.269547 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:27.948554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:27.944015 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:28.762094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:28.762055 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:28.920338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:28.920308 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:30.169106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:30.169072 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:30.489979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:30.489939 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:32.589415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:32.588996 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:33.075516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:33.075481 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:33.224718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:33.224680 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:34.730425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:34.729794 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh"] Oct 29 22:38:34.730425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:34.730053 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" containerID="cri-o://3779faeb6aeaeed935f68dde5deef4ee1ea3930f9c884353fb2c7ebba7bfc833" gracePeriod=30 Oct 29 22:38:34.827074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:34.826126 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll"] Oct 29 22:38:34.827074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:34.826419 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" containerID="cri-o://bda7ba7967d5926c2ee8056e18d1c57a57f70f2b33a86637a2948d152f2bc83b" gracePeriod=30 Oct 29 22:38:34.847506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:34.844889 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47"] Oct 29 22:38:34.847506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:34.845121 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" containerID="cri-o://fc58c540772a8cbec404d24dcce425300a036cdd7e650fb20d73361ce46b3122" gracePeriod=30 Oct 29 22:38:34.862073 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:34.861426 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm"] Oct 29 22:38:34.862073 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:34.861670 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" containerID="cri-o://83308433f77d2127e2a70c9f45b55273c9e4f478da789130bd1676c108ea75d7" gracePeriod=30 Oct 29 22:38:36.041287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:36.041252 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:36.188907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:36.188867 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:36.926965 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:36.926897 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:37.066383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:37.066351 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:37.606679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:37.606640 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:38.068245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:38.068205 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:38:38.068686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:38.068282 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:38:38.742344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:38.742284 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:39.328909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:39.328874 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:39.789206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:39.789168 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:40.208246 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.206766 5610 generic.go:356] "Generic (PLEG): container finished" podID="92f46e04-00bb-4620-872b-cc71207bc368" containerID="bda7ba7967d5926c2ee8056e18d1c57a57f70f2b33a86637a2948d152f2bc83b" exitCode=0 Oct 29 22:38:40.208246 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.206869 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" event={"ID":"92f46e04-00bb-4620-872b-cc71207bc368","Type":"ContainerDied","Data":"bda7ba7967d5926c2ee8056e18d1c57a57f70f2b33a86637a2948d152f2bc83b"} Oct 29 22:38:40.208955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.208932 5610 generic.go:356] "Generic (PLEG): container finished" podID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerID="83308433f77d2127e2a70c9f45b55273c9e4f478da789130bd1676c108ea75d7" exitCode=0 Oct 29 22:38:40.209126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.209111 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" event={"ID":"3ed25461-b10f-43f6-a689-a39a08fc0ac8","Type":"ContainerDied","Data":"83308433f77d2127e2a70c9f45b55273c9e4f478da789130bd1676c108ea75d7"} Oct 29 22:38:40.212248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.212213 5610 generic.go:356] "Generic (PLEG): container finished" podID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerID="fc58c540772a8cbec404d24dcce425300a036cdd7e650fb20d73361ce46b3122" exitCode=0 Oct 29 22:38:40.212360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.212273 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" event={"ID":"eca4a254-88f7-40a2-876c-6dc36cfd9259","Type":"ContainerDied","Data":"fc58c540772a8cbec404d24dcce425300a036cdd7e650fb20d73361ce46b3122"} Oct 29 22:38:40.212360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.212323 5610 scope.go:122] "RemoveContainer" containerID="e34ea2dd206d860bd88724660128aadffa7eb4e64d4da40194e6e13b33de9f63" Oct 29 22:38:40.221861 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.221709 5610 generic.go:356] "Generic (PLEG): container finished" podID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerID="3779faeb6aeaeed935f68dde5deef4ee1ea3930f9c884353fb2c7ebba7bfc833" exitCode=0 Oct 29 22:38:40.221861 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.221771 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" event={"ID":"17df44f9-f686-482d-a4f4-49bb0ba40541","Type":"ContainerDied","Data":"3779faeb6aeaeed935f68dde5deef4ee1ea3930f9c884353fb2c7ebba7bfc833"} Oct 29 22:38:40.221861 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.221816 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" event={"ID":"17df44f9-f686-482d-a4f4-49bb0ba40541","Type":"ContainerDied","Data":"9353660c0d960411ea786089ebae25e11047a17c373b8cad97a3d436d3fb1724"} Oct 29 22:38:40.221861 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.221841 5610 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9353660c0d960411ea786089ebae25e11047a17c373b8cad97a3d436d3fb1724" Oct 29 22:38:40.224172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.223711 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:38:40.225342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.225275 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:38:40.242670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.242596 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.455491 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-data\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.455924 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17df44f9-f686-482d-a4f4-49bb0ba40541-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17df44f9-f686-482d-a4f4-49bb0ba40541-istiod-ca-cert\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.456084 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-socket\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.456131 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-certs\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.456263 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92f46e04-00bb-4620-872b-cc71207bc368-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f46e04-00bb-4620-872b-cc71207bc368-istio-podinfo\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.456432 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-certs\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.456612 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-credential-socket\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.456647 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-data\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.456924 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-kube-api-access-d4qx2\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-kube-api-access-d4qx2\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.456992 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-socket\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.457078 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-envoy\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.457158 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-podinfo\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.458420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.457259 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-envoy\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.463058 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.462853 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-certs" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.473423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.463256 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-data" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.473423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.471410 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-certs" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.473423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.471650 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-socket" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.493373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.473932 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17df44f9-f686-482d-a4f4-49bb0ba40541-istiod-ca-cert" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:40.493373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.487488 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-socket" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.502429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.499552 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-credential-socket" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.573535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.543834 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-data" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.573535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.544667 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:40.670143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.670112 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-istio-token\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-istio-token\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.671588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.671561 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f46e04-00bb-4620-872b-cc71207bc368-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f46e04-00bb-4620-872b-cc71207bc368-istiod-ca-cert\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.671790 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.671774 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-token\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-token\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.671912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.671898 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-credential-socket\") pod \"92f46e04-00bb-4620-872b-cc71207bc368\" (UID: \"92f46e04-00bb-4620-872b-cc71207bc368\") " Oct 29 22:38:40.672029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.672014 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-kube-api-access-xgf2j\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-kube-api-access-xgf2j\") pod \"17df44f9-f686-482d-a4f4-49bb0ba40541\" (UID: \"17df44f9-f686-482d-a4f4-49bb0ba40541\") " Oct 29 22:38:40.673077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.673057 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.676143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.676124 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17df44f9-f686-482d-a4f4-49bb0ba40541-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.676266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.676254 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.676364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.676352 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.676469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.676458 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.676546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.676536 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.676619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.676609 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.676709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.676698 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.686538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.686423 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-envoy" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.686814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.686767 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-credential-socket" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.699834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.698602 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-kube-api-access-d4qx2" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "kube-api-access-d4qx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.701853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.701586 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-envoy" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:40.701853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.701660 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92f46e04-00bb-4620-872b-cc71207bc368-istio-podinfo" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:40.701853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.701712 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-podinfo" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:40.701853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.701803 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-kube-api-access-xgf2j" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "kube-api-access-xgf2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.702657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.702610 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-token" pod "17df44f9-f686-482d-a4f4-49bb0ba40541" (UID: "17df44f9-f686-482d-a4f4-49bb0ba40541"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.703225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.703185 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-istio-token" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:40.710412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.707959 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f46e04-00bb-4620-872b-cc71207bc368-istiod-ca-cert" pod "92f46e04-00bb-4620-872b-cc71207bc368" (UID: "92f46e04-00bb-4620-872b-cc71207bc368"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:40.750562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.749007 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790227 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790262 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790276 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92f46e04-00bb-4620-872b-cc71207bc368-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790288 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790300 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92f46e04-00bb-4620-872b-cc71207bc368-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790315 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgf2j\" (UniqueName: \"kubernetes.io/projected/17df44f9-f686-482d-a4f4-49bb0ba40541-kube-api-access-xgf2j\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790330 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92f46e04-00bb-4620-872b-cc71207bc368-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790346 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qx2\" (UniqueName: \"kubernetes.io/projected/92f46e04-00bb-4620-872b-cc71207bc368-kube-api-access-d4qx2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790359 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.792103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.790372 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17df44f9-f686-482d-a4f4-49bb0ba40541-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:40.984684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:40.976012 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.045294 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-envoy\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.045377 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-socket\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.045445 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-podinfo\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.045511 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-certs\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.045776 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-credential-socket\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.045875 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-token\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-token\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.045932 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca4a254-88f7-40a2-876c-6dc36cfd9259-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca4a254-88f7-40a2-876c-6dc36cfd9259-istiod-ca-cert\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.045962 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-kube-api-access-n8qtt\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-kube-api-access-n8qtt\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.046012 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-data\") pod \"eca4a254-88f7-40a2-876c-6dc36cfd9259\" (UID: \"eca4a254-88f7-40a2-876c-6dc36cfd9259\") " Oct 29 22:38:41.068670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.065475 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-certs" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.070076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.069854 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-data" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.077687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.070327 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca4a254-88f7-40a2-876c-6dc36cfd9259-istiod-ca-cert" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:41.077687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.070847 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eca4a254-88f7-40a2-876c-6dc36cfd9259-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.077687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.070873 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.077687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.070887 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.077687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.074340 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-socket" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.077687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.077307 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-credential-socket" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.078206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.078119 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-envoy" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.083746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.083717 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-podinfo" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:41.083888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.083758 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-kube-api-access-n8qtt" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "kube-api-access-n8qtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:41.084089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.084070 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-token" pod "eca4a254-88f7-40a2-876c-6dc36cfd9259" (UID: "eca4a254-88f7-40a2-876c-6dc36cfd9259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:41.089938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.089610 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:38:41.172185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.171606 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istiod-ca-cert\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.172185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.171680 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-credential-socket\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.172185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.171781 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-certs\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.172185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.171864 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-kube-api-access-dpsjc\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-kube-api-access-dpsjc\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.172185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.171935 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-socket\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.172185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.171936 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istiod-ca-cert" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:38:41.172185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.172095 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-envoy\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.172185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.172157 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-certs" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.172812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.172284 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-credential-socket" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.172812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.172681 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-socket" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.173031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.173004 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-token\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-token\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.173154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.173088 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-podinfo\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.173154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.173143 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-data\") pod \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\" (UID: \"3ed25461-b10f-43f6-a689-a39a08fc0ac8\") " Oct 29 22:38:41.173655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.173634 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-data" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.173980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.173960 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.173980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.173984 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174000 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174013 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8qtt\" (UniqueName: \"kubernetes.io/projected/eca4a254-88f7-40a2-876c-6dc36cfd9259-kube-api-access-n8qtt\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174026 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174038 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174053 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eca4a254-88f7-40a2-876c-6dc36cfd9259-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174065 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eca4a254-88f7-40a2-876c-6dc36cfd9259-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174078 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174090 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174102 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.174976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.174953 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-kube-api-access-dpsjc" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "kube-api-access-dpsjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:41.175620 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.175594 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-podinfo" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:38:41.178160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.178096 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-envoy" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:38:41.179404 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.179373 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-token" pod "3ed25461-b10f-43f6-a689-a39a08fc0ac8" (UID: "3ed25461-b10f-43f6-a689-a39a08fc0ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:38:41.232339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.232313 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" Oct 29 22:38:41.232339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.232327 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll" event={"ID":"92f46e04-00bb-4620-872b-cc71207bc368","Type":"ContainerDied","Data":"bac3a6eb3497255467543861801dc08833a6fda1469e9101c66297bda2a9f737"} Oct 29 22:38:41.232621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.232438 5610 scope.go:122] "RemoveContainer" containerID="bda7ba7967d5926c2ee8056e18d1c57a57f70f2b33a86637a2948d152f2bc83b" Oct 29 22:38:41.233846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.233816 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" Oct 29 22:38:41.233931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.233846 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm" event={"ID":"3ed25461-b10f-43f6-a689-a39a08fc0ac8","Type":"ContainerDied","Data":"de59e460b0970d1f1f2bfe220e1bec0c157ab55ac5d3b48e3b09ca4e0fb5e5cd"} Oct 29 22:38:41.235545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.235484 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" Oct 29 22:38:41.235545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.235530 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47" event={"ID":"eca4a254-88f7-40a2-876c-6dc36cfd9259","Type":"ContainerDied","Data":"c5820f8ee3a3693437abcacb5ccb8ae380367a5dbf8db5ea384f65d6e5b8d3d1"} Oct 29 22:38:41.235700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.235621 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh" Oct 29 22:38:41.276591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.276539 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.276591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.276567 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.276591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.276580 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ed25461-b10f-43f6-a689-a39a08fc0ac8-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.276761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.276594 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpsjc\" (UniqueName: \"kubernetes.io/projected/3ed25461-b10f-43f6-a689-a39a08fc0ac8-kube-api-access-dpsjc\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:38:41.280595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.280566 5610 scope.go:122] "RemoveContainer" containerID="83308433f77d2127e2a70c9f45b55273c9e4f478da789130bd1676c108ea75d7" Oct 29 22:38:41.302547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.302338 5610 scope.go:122] "RemoveContainer" containerID="fc58c540772a8cbec404d24dcce425300a036cdd7e650fb20d73361ce46b3122" Oct 29 22:38:41.336023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.335993 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47"] Oct 29 22:38:41.340927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.340883 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-2zp47"] Oct 29 22:38:41.360361 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:38:41.360323 5610 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed25461_b10f_43f6_a689_a39a08fc0ac8.slice/crio-de59e460b0970d1f1f2bfe220e1bec0c157ab55ac5d3b48e3b09ca4e0fb5e5cd\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed25461_b10f_43f6_a689_a39a08fc0ac8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17df44f9_f686_482d_a4f4_49bb0ba40541.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17df44f9_f686_482d_a4f4_49bb0ba40541.slice/crio-9353660c0d960411ea786089ebae25e11047a17c373b8cad97a3d436d3fb1724\": RecentStats: unable to find data in memory cache]" Oct 29 22:38:41.372268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.372238 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm"] Oct 29 22:38:41.378149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.378115 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-wzzjm"] Oct 29 22:38:41.463951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.463916 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll"] Oct 29 22:38:41.507315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.507226 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-jcvll"] Oct 29 22:38:41.531478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.531442 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh"] Oct 29 22:38:41.534476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.534442 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4vtrh"] Oct 29 22:38:41.644155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.644116 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" path="/var/lib/kubelet/pods/17df44f9-f686-482d-a4f4-49bb0ba40541/volumes" Oct 29 22:38:41.645277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.645253 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" path="/var/lib/kubelet/pods/3ed25461-b10f-43f6-a689-a39a08fc0ac8/volumes" Oct 29 22:38:41.646231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.646210 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f46e04-00bb-4620-872b-cc71207bc368" path="/var/lib/kubelet/pods/92f46e04-00bb-4620-872b-cc71207bc368/volumes" Oct 29 22:38:41.647750 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:41.647723 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" path="/var/lib/kubelet/pods/eca4a254-88f7-40a2-876c-6dc36cfd9259/volumes" Oct 29 22:38:42.221617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:42.221578 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:43.713921 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:38:43.713886 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:38:43.906165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:43.906128 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:44.231426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:44.231368 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:44.464414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:44.462102 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:45.699344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.699181 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:45.773252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.773216 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx"] Oct 29 22:38:45.774113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774093 5610 cpu_manager.go:414] "containerMap: removing container" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774113 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774149 5610 cpu_manager.go:414] "containerMap: removing container" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774159 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774194 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774201 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774221 5610 cpu_manager.go:414] "containerMap: removing container" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774228 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774243 5610 cpu_manager.go:414] "containerMap: removing container" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" Oct 29 22:38:45.774256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774249 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" Oct 29 22:38:45.774700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774458 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="17df44f9-f686-482d-a4f4-49bb0ba40541" containerName="istio-proxy" Oct 29 22:38:45.774700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774477 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed25461-b10f-43f6-a689-a39a08fc0ac8" containerName="istio-proxy" Oct 29 22:38:45.774700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774502 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" Oct 29 22:38:45.774700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774512 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f46e04-00bb-4620-872b-cc71207bc368" containerName="istio-proxy" Oct 29 22:38:45.774700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.774532 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca4a254-88f7-40a2-876c-6dc36cfd9259" containerName="istio-proxy" Oct 29 22:38:45.778644 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.778615 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:45.793403 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.793328 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e526ec\"" Oct 29 22:38:45.810757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.810724 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx"] Oct 29 22:38:45.834656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.834607 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9"] Oct 29 22:38:45.839365 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.839335 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:45.844441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.844379 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bf7424-dda4-459d-b823-26355340ffdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx\" (UID: \"52bf7424-dda4-459d-b823-26355340ffdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:45.844777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.844754 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c52v9\" (UniqueName: \"kubernetes.io/projected/52bf7424-dda4-459d-b823-26355340ffdc-kube-api-access-c52v9\") pod \"payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx\" (UID: \"52bf7424-dda4-459d-b823-26355340ffdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:45.862167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.862130 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9"] Oct 29 22:38:45.864113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.864068 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:45.898414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.886498 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e526ec-dockercfg-lkrjl\"" Oct 29 22:38:45.997577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:45.973273 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glpr5\" (UniqueName: \"kubernetes.io/projected/ab96daa8-8158-4802-8084-81af3befaedf-kube-api-access-glpr5\") pod \"payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9\" (UID: \"ab96daa8-8158-4802-8084-81af3befaedf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:46.011564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.011522 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab96daa8-8158-4802-8084-81af3befaedf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9\" (UID: \"ab96daa8-8158-4802-8084-81af3befaedf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:46.011937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.011911 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c52v9\" (UniqueName: \"kubernetes.io/projected/52bf7424-dda4-459d-b823-26355340ffdc-kube-api-access-c52v9\") pod \"payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx\" (UID: \"52bf7424-dda4-459d-b823-26355340ffdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:46.012340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.012317 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bf7424-dda4-459d-b823-26355340ffdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx\" (UID: \"52bf7424-dda4-459d-b823-26355340ffdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:46.033425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.013464 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bf7424-dda4-459d-b823-26355340ffdc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx\" (UID: \"52bf7424-dda4-459d-b823-26355340ffdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:46.120712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.071194 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c52v9\" (UniqueName: \"kubernetes.io/projected/52bf7424-dda4-459d-b823-26355340ffdc-kube-api-access-c52v9\") pod \"payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx\" (UID: \"52bf7424-dda4-459d-b823-26355340ffdc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:46.167562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.140658 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:46.186423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.170281 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glpr5\" (UniqueName: \"kubernetes.io/projected/ab96daa8-8158-4802-8084-81af3befaedf-kube-api-access-glpr5\") pod \"payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9\" (UID: \"ab96daa8-8158-4802-8084-81af3befaedf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:46.195585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.195546 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab96daa8-8158-4802-8084-81af3befaedf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9\" (UID: \"ab96daa8-8158-4802-8084-81af3befaedf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:46.246969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.208999 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab96daa8-8158-4802-8084-81af3befaedf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9\" (UID: \"ab96daa8-8158-4802-8084-81af3befaedf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:46.417973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.417937 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:46.478574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.478504 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glpr5\" (UniqueName: \"kubernetes.io/projected/ab96daa8-8158-4802-8084-81af3befaedf-kube-api-access-glpr5\") pod \"payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9\" (UID: \"ab96daa8-8158-4802-8084-81af3befaedf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:46.591638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.591525 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:46.764413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.757691 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:46.991275 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:46.991244 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx"] Oct 29 22:38:46.992636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:38:46.992601 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bf7424_dda4_459d_b823_26355340ffdc.slice/crio-4e3aabef22851d97c787142fcb866408aed91c73b1308245b1196975ae6b116b WatchSource:0}: Error finding container 4e3aabef22851d97c787142fcb866408aed91c73b1308245b1196975ae6b116b: Status 404 returned error can't find the container with id 4e3aabef22851d97c787142fcb866408aed91c73b1308245b1196975ae6b116b Oct 29 22:38:47.077901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.077865 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9"] Oct 29 22:38:47.078314 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:38:47.078286 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab96daa8_8158_4802_8084_81af3befaedf.slice/crio-8f168360a699491033dcef4096d6e831ddd8c351e56428ccb327829106f620b4 WatchSource:0}: Error finding container 8f168360a699491033dcef4096d6e831ddd8c351e56428ccb327829106f620b4: Status 404 returned error can't find the container with id 8f168360a699491033dcef4096d6e831ddd8c351e56428ccb327829106f620b4 Oct 29 22:38:47.537688 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.537647 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" event={"ID":"52bf7424-dda4-459d-b823-26355340ffdc","Type":"ContainerStarted","Data":"56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0"} Oct 29 22:38:47.537900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.537713 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" event={"ID":"52bf7424-dda4-459d-b823-26355340ffdc","Type":"ContainerStarted","Data":"4e3aabef22851d97c787142fcb866408aed91c73b1308245b1196975ae6b116b"} Oct 29 22:38:47.537900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.537860 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:47.539587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.539563 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" event={"ID":"ab96daa8-8158-4802-8084-81af3befaedf","Type":"ContainerStarted","Data":"f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c"} Oct 29 22:38:47.539699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.539605 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" event={"ID":"ab96daa8-8158-4802-8084-81af3befaedf","Type":"ContainerStarted","Data":"8f168360a699491033dcef4096d6e831ddd8c351e56428ccb327829106f620b4"} Oct 29 22:38:47.539699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.539629 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:47.577758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.577097 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" podStartSLOduration=2.577078292 podStartE2EDuration="2.577078292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:47.577034803 +0000 UTC m=+1390.058352298" watchObservedRunningTime="2026-10-29 22:38:47.577078292 +0000 UTC m=+1390.058395786" Oct 29 22:38:47.615204 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:47.615135 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" podStartSLOduration=2.615117414 podStartE2EDuration="2.615117414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:38:47.614186509 +0000 UTC m=+1390.095504001" watchObservedRunningTime="2026-10-29 22:38:47.615117414 +0000 UTC m=+1390.096434908" Oct 29 22:38:48.542418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:48.539974 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:50.041659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:50.039811 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:51.692349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:51.690814 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:53.158418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:53.158326 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:54.719889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:54.719852 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:56.602593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:56.602558 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:58.258416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:58.257934 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:38:58.609781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:58.609691 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:38:58.609933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:58.609824 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:38:59.800416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:38:59.794021 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:01.937741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:01.937697 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk"] Oct 29 22:39:01.943998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:01.943972 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:01.968773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:01.968602 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a26426-serving-cert\"" Oct 29 22:39:01.975312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:01.975281 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk"] Oct 29 22:39:02.062690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.062654 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.062905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.062713 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-metrics-certs\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.063054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.063022 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd032-fc16-4969-9144-9f030202ca86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.063133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.063113 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hvw\" (UniqueName: \"kubernetes.io/projected/10fcd032-fc16-4969-9144-9f030202ca86-kube-api-access-q5hvw\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.069636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.069604 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:02.185758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.185709 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd032-fc16-4969-9144-9f030202ca86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.185975 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.185830 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hvw\" (UniqueName: \"kubernetes.io/projected/10fcd032-fc16-4969-9144-9f030202ca86-kube-api-access-q5hvw\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.185975 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.185964 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.186099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.185998 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-metrics-certs\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.187868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.187795 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd032-fc16-4969-9144-9f030202ca86-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.188348 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:02.188317 5610 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-serving-cert: secret "maas-api-e2e-inf-a-a26426-serving-cert" not found Oct 29 22:39:02.201415 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:02.188585 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls podName:10fcd032-fc16-4969-9144-9f030202ca86 nodeName:}" failed. No retries permitted until 2026-10-29 22:39:02.688563902 +0000 UTC m=+1405.169881387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls") pod "maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" (UID: "10fcd032-fc16-4969-9144-9f030202ca86") : secret "maas-api-e2e-inf-a-a26426-serving-cert" not found Oct 29 22:39:02.210458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.210424 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-metrics-certs\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.236412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.228406 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:02.236412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.231889 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hvw\" (UniqueName: \"kubernetes.io/projected/10fcd032-fc16-4969-9144-9f030202ca86-kube-api-access-q5hvw\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.399749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.399710 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:02.698879 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.698823 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.701585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.701555 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls\") pod \"maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:02.871427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:02.864715 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:03.275936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.275897 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk"] Oct 29 22:39:03.826197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.826160 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt"] Oct 29 22:39:03.833066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.833040 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.844298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.844271 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7"] Oct 29 22:39:03.848618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.848591 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0ab33e-openshift-default-dockercfg-gm29v\"" Oct 29 22:39:03.849383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.849366 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.859333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859301 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.859480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859434 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.859480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859472 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.859617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859527 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.859617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859580 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.859617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859610 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5945809b-9330-4c21-a1df-47e63d06ec59-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.859787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859754 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.859848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859806 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.859904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859844 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.859904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859869 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.860014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859937 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5945809b-9330-4c21-a1df-47e63d06ec59-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.860014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.859984 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.860014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.860009 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.860179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.860034 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.860179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.860055 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.870712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.869423 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.870712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.869489 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5lpd\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-kube-api-access-m5lpd\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.870712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.869554 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcggv\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-kube-api-access-tcggv\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.873478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.873450 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt"] Oct 29 22:39:03.915520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.911492 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7"] Oct 29 22:39:03.929436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.929339 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" event={"ID":"10fcd032-fc16-4969-9144-9f030202ca86","Type":"ContainerStarted","Data":"eb8689a1987291e5fb99ed295cfa6edae5e903b16d064f9fca244bf05aec6c38"} Oct 29 22:39:03.971899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.971843 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.972147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.971974 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.972147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972060 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.972147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972142 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.972388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972173 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5945809b-9330-4c21-a1df-47e63d06ec59-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.972388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972236 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.972388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972309 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.972388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972384 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.972388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972435 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.972844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972526 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5945809b-9330-4c21-a1df-47e63d06ec59-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.972844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972621 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.972844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972683 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.972844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972725 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.972844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972758 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.972844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972807 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.973276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972853 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5lpd\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-kube-api-access-m5lpd\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.973276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.972918 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcggv\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-kube-api-access-tcggv\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.973276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.973084 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.974093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.974069 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.974463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.974420 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-certs\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.974815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.974794 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.975145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.975122 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.976721 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.976695 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.977038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.977016 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-data\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.977267 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.977245 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.977996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.977964 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.978462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.978424 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5945809b-9330-4c21-a1df-47e63d06ec59-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.979418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.979345 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5945809b-9330-4c21-a1df-47e63d06ec59-istiod-ca-cert\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.980098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.980077 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-credential-socket\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.981434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.981378 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.983415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.983134 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-podinfo\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.983415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.983287 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-envoy\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.992023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.991998 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:03.997237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.997209 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcggv\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-kube-api-access-tcggv\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:03.998759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:03.998735 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5lpd\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-kube-api-access-m5lpd\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:04.000084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.000041 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-token\") pod \"e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:04.191467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.191343 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:04.207803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.207774 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:04.253247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.251599 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:04.549984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.532206 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:04.693960 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.693895 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:04.828514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.810435 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt"] Oct 29 22:39:04.885001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.884963 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:04.885152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.885052 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:04.885152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.885098 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:04.965768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.965686 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7"] Oct 29 22:39:04.979132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.979097 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" event={"ID":"175f111a-fe47-40c6-bdf3-4b94f172ccfa","Type":"ContainerStarted","Data":"5b9aff9ddb7039c708ecda969856f118a1e18cad429e8469b062820f2a6b3a21"} Oct 29 22:39:04.981824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.981795 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" event={"ID":"10fcd032-fc16-4969-9144-9f030202ca86","Type":"ContainerStarted","Data":"267919014d320012d0f6b0d4b1a015ba668e732b08668db183897218a2e88dd9"} Oct 29 22:39:04.982318 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.982297 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:04.988078 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:39:04.988046 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5945809b_9330_4c21_a1df_47e63d06ec59.slice/crio-4c9ee2169c58c17429ecd0429b63964c03a791c2096576068dbd31534946ecb5 WatchSource:0}: Error finding container 4c9ee2169c58c17429ecd0429b63964c03a791c2096576068dbd31534946ecb5: Status 404 returned error can't find the container with id 4c9ee2169c58c17429ecd0429b63964c03a791c2096576068dbd31534946ecb5 Oct 29 22:39:04.992355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.992328 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:04.992550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.992532 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:04.992674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:04.992647 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:05.027408 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:05.027351 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" podStartSLOduration=3.559615777 podStartE2EDuration="4.027334992s" totalImagesPullingTime="467.719215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:05.023534218 +0000 UTC m=+1407.504851712" watchObservedRunningTime="2026-10-29 22:39:05.027334992 +0000 UTC m=+1407.508652485" Oct 29 22:39:05.121554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:05.121506 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:05.849413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:05.842910 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:06.034608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.034571 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:06.249596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.245195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:06.442780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.442743 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" event={"ID":"175f111a-fe47-40c6-bdf3-4b94f172ccfa","Type":"ContainerStarted","Data":"9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79"} Oct 29 22:39:06.447899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.447726 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" event={"ID":"5945809b-9330-4c21-a1df-47e63d06ec59","Type":"ContainerStarted","Data":"54ddf99a0440c99192d0ed50bc6577c4964f7469fbe50379f18fca0647cce1fa"} Oct 29 22:39:06.448302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.448271 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" event={"ID":"5945809b-9330-4c21-a1df-47e63d06ec59","Type":"ContainerStarted","Data":"4c9ee2169c58c17429ecd0429b63964c03a791c2096576068dbd31534946ecb5"} Oct 29 22:39:06.620471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.528205 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:06.620471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.535077 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podStartSLOduration=3.535056216 podStartE2EDuration="3.535056216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:06.526266115 +0000 UTC m=+1409.007583608" watchObservedRunningTime="2026-10-29 22:39:06.535056216 +0000 UTC m=+1409.016373711" Oct 29 22:39:06.620471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.604086 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podStartSLOduration=3.60406708 podStartE2EDuration="3.60406708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:06.603738522 +0000 UTC m=+1409.085056015" watchObservedRunningTime="2026-10-29 22:39:06.60406708 +0000 UTC m=+1409.085384578" Oct 29 22:39:06.922633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:06.912024 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:07.196528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:07.194052 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:07.196528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:07.194576 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:07.224497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:07.212781 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:39:07.224497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:07.212847 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 29 22:39:07.236956 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:07.236827 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:07.236956 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:07.236886 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 29 22:39:07.237228 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:07.237205 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:07.604855 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:07.604763 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:08.193858 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:08.193818 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:39:08.194496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:08.194461 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 29 22:39:08.216337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:08.216300 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:08.216648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:08.216620 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 29 22:39:09.229089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:09.223525 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:39:09.229089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:09.223597 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 29 22:39:09.232107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:09.232005 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:09.232107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:09.232060 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 29 22:39:10.195416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:10.194778 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:39:10.195416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:10.194856 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 29 22:39:10.224417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:10.219324 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:10.224417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:10.219381 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 29 22:39:10.521475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:10.520062 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:11.206417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:11.203300 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:39:11.206417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:11.203374 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 29 22:39:11.226467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:11.225773 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:11.226467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:11.225837 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 29 22:39:11.452715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:11.452685 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:39:12.204418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:12.204169 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.204418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:12.204233 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 29 22:39:12.219427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:12.210807 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:12.219427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:12.210866 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 29 22:39:12.383418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:12.381603 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:12.927023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:12.926975 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:13.091242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:13.091195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:13.196919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:13.196883 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.197190 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:13.197160 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 29 22:39:13.211746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:13.211711 5610 patch_prober.go:29] interesting pod/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 29 22:39:13.212315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:13.212281 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 29 22:39:13.583481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:13.575922 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:13.984624 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:13.984582 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:39:14.197274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:14.197246 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:14.197662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:14.197638 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:14.199266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:14.199246 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:39:14.307243 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:14.304666 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:14.307243 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:14.305829 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:14.312432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:14.312411 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:39:14.516308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:14.516261 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:14.918645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:14.918602 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:16.880904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:16.744630 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:18.323339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:18.323305 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:19.937314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:19.936451 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:21.753236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:21.753202 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:21.929415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:21.925928 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:22.158761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:22.158679 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:23.570078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:23.570036 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:25.046134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:25.046094 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:26.584419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:26.568946 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:28.292731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:28.292696 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:29.951872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:29.951840 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:31.431083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:31.431048 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:33.246379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:33.246342 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:35.379535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:35.379493 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:36.862907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:36.862873 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:37.696737 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:37.696701 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:38.311839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:38.311800 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:39.657232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:39.657203 5610 scope.go:122] "RemoveContainer" containerID="3779faeb6aeaeed935f68dde5deef4ee1ea3930f9c884353fb2c7ebba7bfc833" Oct 29 22:39:40.037196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:40.037157 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:41.594414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:41.590936 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:42.128744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.128705 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9020-b267e7"] Oct 29 22:39:42.136032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.135995 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9020-b267e7" Oct 29 22:39:42.155853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.154426 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9020-b267e7"] Oct 29 22:39:42.156743 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.156714 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c5k5w\"" Oct 29 22:39:42.203764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.203727 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xd9\" (UniqueName: \"kubernetes.io/projected/2c1bfef4-a428-46c3-a590-e584c9593b6a-kube-api-access-h9xd9\") pod \"test-curl-9020-b267e7\" (UID: \"2c1bfef4-a428-46c3-a590-e584c9593b6a\") " pod="opendatahub/test-curl-9020-b267e7" Oct 29 22:39:42.313415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.312860 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xd9\" (UniqueName: \"kubernetes.io/projected/2c1bfef4-a428-46c3-a590-e584c9593b6a-kube-api-access-h9xd9\") pod \"test-curl-9020-b267e7\" (UID: \"2c1bfef4-a428-46c3-a590-e584c9593b6a\") " pod="opendatahub/test-curl-9020-b267e7" Oct 29 22:39:42.408060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.407979 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xd9\" (UniqueName: \"kubernetes.io/projected/2c1bfef4-a428-46c3-a590-e584c9593b6a-kube-api-access-h9xd9\") pod \"test-curl-9020-b267e7\" (UID: \"2c1bfef4-a428-46c3-a590-e584c9593b6a\") " pod="opendatahub/test-curl-9020-b267e7" Oct 29 22:39:42.454092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.454055 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9020-b267e7" Oct 29 22:39:42.774595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:42.774562 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9020-b267e7"] Oct 29 22:39:42.782692 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:39:42.782659 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c1bfef4_a428_46c3_a590_e584c9593b6a.slice/crio-cec6391a14728094c0a50573a50f84e2dd0eb8fb2090bc3f5527e0e6eb7f0869 WatchSource:0}: Error finding container cec6391a14728094c0a50573a50f84e2dd0eb8fb2090bc3f5527e0e6eb7f0869: Status 404 returned error can't find the container with id cec6391a14728094c0a50573a50f84e2dd0eb8fb2090bc3f5527e0e6eb7f0869 Oct 29 22:39:43.503963 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:43.502474 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9020-b267e7" event={"ID":"2c1bfef4-a428-46c3-a590-e584c9593b6a","Type":"ContainerStarted","Data":"cec6391a14728094c0a50573a50f84e2dd0eb8fb2090bc3f5527e0e6eb7f0869"} Oct 29 22:39:43.736575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:43.716739 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:43.815483 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:43.814635 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:39:44.635878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:44.635837 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:44.777936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:44.777900 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:45.401235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:45.401047 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:45.649329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:45.649287 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:45.818279 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:45.818247 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:45.985529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:45.985432 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:46.129575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:46.129495 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:46.284419 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:46.279177 5610 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c1bfef4_a428_46c3_a590_e584c9593b6a.slice/crio-447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:39:46.345478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:46.331786 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:46.665428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:46.665365 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:47.097350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.092498 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:47.135417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.133453 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r"] Oct 29 22:39:47.241226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.238290 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:47.252732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.252532 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r"] Oct 29 22:39:47.258547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.258502 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-433712\"" Oct 29 22:39:47.285944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.285895 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-433712-dockercfg-pg6rt\"" Oct 29 22:39:47.317589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.317547 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jthf\" (UniqueName: \"kubernetes.io/projected/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-kube-api-access-7jthf\") pod \"payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r\" (UID: \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:47.320867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.320197 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r\" (UID: \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:47.434472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.428518 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jthf\" (UniqueName: \"kubernetes.io/projected/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-kube-api-access-7jthf\") pod \"payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r\" (UID: \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:47.434472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.428728 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r\" (UID: \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:47.434472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.429939 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r\" (UID: \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:47.476262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.473904 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jthf\" (UniqueName: \"kubernetes.io/projected/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-kube-api-access-7jthf\") pod \"payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r\" (UID: \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:47.591017 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.590974 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:47.799573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.799535 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9020-b267e7" event={"ID":"2c1bfef4-a428-46c3-a590-e584c9593b6a","Type":"ContainerStarted","Data":"447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff"} Oct 29 22:39:47.840570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.840510 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9020-b267e7" podStartSLOduration=2.458945016 podStartE2EDuration="5.840496007s" totalImagesPullingTime="3.381550991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:47.834844561 +0000 UTC m=+1450.316162059" watchObservedRunningTime="2026-10-29 22:39:47.840496007 +0000 UTC m=+1450.321813497" Oct 29 22:39:47.902003 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:39:47.901971 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ea960c_2e98_4332_bb5a_6a6b38f9d63c.slice/crio-bda80c7dc6df271c85c061aa6cb6c0bcd3bc1cb08a4d38f56508e3873082a3cb WatchSource:0}: Error finding container bda80c7dc6df271c85c061aa6cb6c0bcd3bc1cb08a4d38f56508e3873082a3cb: Status 404 returned error can't find the container with id bda80c7dc6df271c85c061aa6cb6c0bcd3bc1cb08a4d38f56508e3873082a3cb Oct 29 22:39:47.902700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:47.902666 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r"] Oct 29 22:39:48.397935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:48.397896 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:48.543603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:48.543575 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:48.942226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:48.940652 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:49.148242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.141772 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" event={"ID":"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c","Type":"ContainerStarted","Data":"7408d96c74f6ac4649c68ce0ed3d012ae2a6c8b57314ed3abbb9d38f45156b6d"} Oct 29 22:39:49.148242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.141881 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" event={"ID":"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c","Type":"ContainerStarted","Data":"bda80c7dc6df271c85c061aa6cb6c0bcd3bc1cb08a4d38f56508e3873082a3cb"} Oct 29 22:39:49.148242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.142304 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:39:49.148242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.146967 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9020-b267e7"] Oct 29 22:39:49.153350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.153280 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9020-b267e7"] Oct 29 22:39:49.186677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.185429 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" podStartSLOduration=3.185410521 podStartE2EDuration="3.185410521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:49.179508708 +0000 UTC m=+1451.660826197" watchObservedRunningTime="2026-10-29 22:39:49.185410521 +0000 UTC m=+1451.666728016" Oct 29 22:39:49.683414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.668561 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk"] Oct 29 22:39:49.683414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.668817 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" containerID="cri-o://d01c5755e5f00609f395119461e1ba37dd2d76c715c2d27a2ec7524b19db22ca" gracePeriod=30 Oct 29 22:39:49.683414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.680424 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b"] Oct 29 22:39:49.683414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.680700 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" containerID="cri-o://04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463" gracePeriod=30 Oct 29 22:39:49.690542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.690510 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx"] Oct 29 22:39:49.691445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.690757 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" containerID="cri-o://bdd0e9146b6f091fd3efa903052377e1da705dd6981cfa93da504f47803949a4" gracePeriod=30 Oct 29 22:39:49.700567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.700536 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc"] Oct 29 22:39:49.700966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.700938 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" containerID="cri-o://16db3dc1ec23bd55692695596d10f64a286cb3db6deb7dd965bcb878dd76f736" gracePeriod=30 Oct 29 22:39:49.709537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.709180 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk"] Oct 29 22:39:49.709537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.709425 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" containerID="cri-o://46bd259206ee3467dc4e64e7c803b98db1c684fb69456dbf88c739204d0f9d20" gracePeriod=30 Oct 29 22:39:49.717157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.716607 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9"] Oct 29 22:39:49.722937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.722912 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg"] Oct 29 22:39:49.723148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.723110 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" containerID="cri-o://52d035c53772f65f13487294cb042e7097a061bc150199f45157d10a2d2455e4" gracePeriod=30 Oct 29 22:39:49.730439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.728448 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97"] Oct 29 22:39:49.730439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:49.728675 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" containerID="cri-o://c97f5a6f53159efd9f0210794f0704b4bd7a27fe5aa9e5293d8bfd2af8ac2c9b" gracePeriod=30 Oct 29 22:39:50.148850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:50.148806 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" containerID="cri-o://4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b" gracePeriod=30 Oct 29 22:39:50.149357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:50.149033 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9020-b267e7" podUID="2c1bfef4-a428-46c3-a590-e584c9593b6a" containerName="test-curl-9020-b267e7" containerID="cri-o://447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff" gracePeriod=2 Oct 29 22:39:50.820073 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:50.820028 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:52.408408 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:52.408327 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9020-b267e7" Oct 29 22:39:52.465938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:52.465907 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1bfef4-a428-46c3-a590-e584c9593b6a-kube-api-access-h9xd9\" (UniqueName: \"kubernetes.io/projected/2c1bfef4-a428-46c3-a590-e584c9593b6a-kube-api-access-h9xd9\") pod \"2c1bfef4-a428-46c3-a590-e584c9593b6a\" (UID: \"2c1bfef4-a428-46c3-a590-e584c9593b6a\") " Oct 29 22:39:52.468101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:52.468065 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1bfef4-a428-46c3-a590-e584c9593b6a-kube-api-access-h9xd9" pod "2c1bfef4-a428-46c3-a590-e584c9593b6a" (UID: "2c1bfef4-a428-46c3-a590-e584c9593b6a"). InnerVolumeSpecName "kube-api-access-h9xd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:52.568106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:52.568069 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xd9\" (UniqueName: \"kubernetes.io/projected/2c1bfef4-a428-46c3-a590-e584c9593b6a-kube-api-access-h9xd9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:52.621465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:52.621415 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:53.176498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:53.176456 5610 generic.go:356] "Generic (PLEG): container finished" podID="2c1bfef4-a428-46c3-a590-e584c9593b6a" containerID="447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff" exitCode=137 Oct 29 22:39:53.176679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:53.176523 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9020-b267e7" Oct 29 22:39:53.176679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:53.176596 5610 scope.go:122] "RemoveContainer" containerID="447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff" Oct 29 22:39:53.189209 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:53.189187 5610 scope.go:122] "RemoveContainer" containerID="447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff" Oct 29 22:39:53.189560 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:53.189536 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff\": container with ID starting with 447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff not found: ID does not exist" containerID="447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff" Oct 29 22:39:53.189637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:53.189573 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff"} err="failed to get container status \"447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff\": rpc error: code = NotFound desc = could not find container \"447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff\": container with ID starting with 447fe47495ab1250b2185a43803c801d3bf1f80cf0a2b71dfaa85f9b15c605ff not found: ID does not exist" Oct 29 22:39:53.198566 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:53.198535 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9020-b267e7\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="2c1bfef4-a428-46c3-a590-e584c9593b6a" pod="opendatahub/test-curl-9020-b267e7" Oct 29 22:39:53.639317 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:53.639280 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1bfef4-a428-46c3-a590-e584c9593b6a" path="/var/lib/kubelet/pods/2c1bfef4-a428-46c3-a590-e584c9593b6a/volumes" Oct 29 22:39:54.247292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:54.247259 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:55.086165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.086135 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175031 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-kube-api-access-5nc2v\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-kube-api-access-5nc2v\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175098 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-credential-socket\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175196 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istiod-ca-cert\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175230 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-envoy\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175261 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-podinfo\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175364 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-certs\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175418 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-socket\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175468 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-data\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.175624 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-token\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-token\") pod \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\" (UID: \"d7084709-6ae5-4bba-b2b1-6c590b3ee21d\") " Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.181514 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-credential-socket" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.194053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.190922 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-certs" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.196704 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istiod-ca-cert" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.197719 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-data" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.198067 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-envoy" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.201646 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-socket" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.204607 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.204630 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.204644 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.204657 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.204669 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.209138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.204682 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.218524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.217574 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-kube-api-access-5nc2v" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "kube-api-access-5nc2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.224418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.222251 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms"] Oct 29 22:39:55.224418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.223559 5610 cpu_manager.go:414] "containerMap: removing container" podUID="2c1bfef4-a428-46c3-a590-e584c9593b6a" containerName="test-curl-9020-b267e7" Oct 29 22:39:55.224418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.223577 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1bfef4-a428-46c3-a590-e584c9593b6a" containerName="test-curl-9020-b267e7" Oct 29 22:39:55.224418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.223597 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" Oct 29 22:39:55.224418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.223606 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" Oct 29 22:39:55.224418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.223820 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1bfef4-a428-46c3-a590-e584c9593b6a" containerName="test-curl-9020-b267e7" Oct 29 22:39:55.224418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.223835 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerName="istio-proxy" Oct 29 22:39:55.230830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.230809 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms"] Oct 29 22:39:55.230951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.230897 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.231024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.231006 5610 generic.go:356] "Generic (PLEG): container finished" podID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerID="d01c5755e5f00609f395119461e1ba37dd2d76c715c2d27a2ec7524b19db22ca" exitCode=0 Oct 29 22:39:55.231158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.231106 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" event={"ID":"9ae8da64-61ec-463f-b44f-299bd4c6d1b4","Type":"ContainerDied","Data":"d01c5755e5f00609f395119461e1ba37dd2d76c715c2d27a2ec7524b19db22ca"} Oct 29 22:39:55.236050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.236006 5610 generic.go:356] "Generic (PLEG): container finished" podID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerID="52d035c53772f65f13487294cb042e7097a061bc150199f45157d10a2d2455e4" exitCode=0 Oct 29 22:39:55.236873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.236171 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" event={"ID":"0341c0e9-195b-4953-bcf4-e5f34f7949d9","Type":"ContainerDied","Data":"52d035c53772f65f13487294cb042e7097a061bc150199f45157d10a2d2455e4"} Oct 29 22:39:55.236873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.236830 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-token" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.238759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.238584 5610 generic.go:356] "Generic (PLEG): container finished" podID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" containerID="04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463" exitCode=0 Oct 29 22:39:55.238759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.238731 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" Oct 29 22:39:55.238917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.238779 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" event={"ID":"d7084709-6ae5-4bba-b2b1-6c590b3ee21d","Type":"ContainerDied","Data":"04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463"} Oct 29 22:39:55.238917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.238831 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b" event={"ID":"d7084709-6ae5-4bba-b2b1-6c590b3ee21d","Type":"ContainerDied","Data":"3688b380303b7fdc00674bbecda26c92f98c9275c46dd5cda863d6f9a29edac7"} Oct 29 22:39:55.238917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.238861 5610 scope.go:122] "RemoveContainer" containerID="04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463" Oct 29 22:39:55.241168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.240978 5610 generic.go:356] "Generic (PLEG): container finished" podID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerID="46bd259206ee3467dc4e64e7c803b98db1c684fb69456dbf88c739204d0f9d20" exitCode=0 Oct 29 22:39:55.241168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.241055 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" event={"ID":"d50172ea-80ec-49ee-8534-f8f9841a4fc9","Type":"ContainerDied","Data":"46bd259206ee3467dc4e64e7c803b98db1c684fb69456dbf88c739204d0f9d20"} Oct 29 22:39:55.247832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.246451 5610 generic.go:356] "Generic (PLEG): container finished" podID="4daf7ff6-9658-4276-90f2-a148d32143af" containerID="16db3dc1ec23bd55692695596d10f64a286cb3db6deb7dd965bcb878dd76f736" exitCode=0 Oct 29 22:39:55.247832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.246558 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" event={"ID":"4daf7ff6-9658-4276-90f2-a148d32143af","Type":"ContainerDied","Data":"16db3dc1ec23bd55692695596d10f64a286cb3db6deb7dd965bcb878dd76f736"} Oct 29 22:39:55.247832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.246897 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-6bb06a-openshift-default-dockercfg-vhrww\"" Oct 29 22:39:55.299513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.298769 5610 generic.go:356] "Generic (PLEG): container finished" podID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerID="c97f5a6f53159efd9f0210794f0704b4bd7a27fe5aa9e5293d8bfd2af8ac2c9b" exitCode=0 Oct 29 22:39:55.299513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.298920 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" event={"ID":"884ab263-fded-4dec-b6f5-bb9799fbb140","Type":"ContainerDied","Data":"c97f5a6f53159efd9f0210794f0704b4bd7a27fe5aa9e5293d8bfd2af8ac2c9b"} Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.305220 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-podinfo" pod "d7084709-6ae5-4bba-b2b1-6c590b3ee21d" (UID: "d7084709-6ae5-4bba-b2b1-6c590b3ee21d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306282 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306360 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306521 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5h76\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-kube-api-access-z5h76\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306577 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306609 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306654 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306780 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306847 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.306938 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.307015 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.307030 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nc2v\" (UniqueName: \"kubernetes.io/projected/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-kube-api-access-5nc2v\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.307044 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7084709-6ae5-4bba-b2b1-6c590b3ee21d-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.309474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.308384 5610 scope.go:122] "RemoveContainer" containerID="04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463" Oct 29 22:39:55.310660 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:55.309607 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463\": container with ID starting with 04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463 not found: ID does not exist" containerID="04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463" Oct 29 22:39:55.310660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.309639 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463"} err="failed to get container status \"04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463\": rpc error: code = NotFound desc = could not find container \"04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463\": container with ID starting with 04b60fb15c283cdbae5c5d7c678ba53652bbee5caa8ade7377ceb0bcc4878463 not found: ID does not exist" Oct 29 22:39:55.327163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.327027 5610 generic.go:356] "Generic (PLEG): container finished" podID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerID="bdd0e9146b6f091fd3efa903052377e1da705dd6981cfa93da504f47803949a4" exitCode=0 Oct 29 22:39:55.327163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.327093 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" event={"ID":"aa15275f-8808-4e9a-bb73-875dedce34c6","Type":"ContainerDied","Data":"bdd0e9146b6f091fd3efa903052377e1da705dd6981cfa93da504f47803949a4"} Oct 29 22:39:55.331815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.330938 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:39:55.339721 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.338356 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:39:55.346485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.346465 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:39:55.352100 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.352049 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:39:55.357641 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.357627 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:39:55.364577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.364561 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:39:55.435635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.413468 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa15275f-8808-4e9a-bb73-875dedce34c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa15275f-8808-4e9a-bb73-875dedce34c6-istiod-ca-cert\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.435817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.435640 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-data\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.435817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.435698 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-token\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-token\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.435817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.435771 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-certs\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.435983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.435815 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-socket\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.435983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.435849 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-kube-api-access-sx2qw\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-kube-api-access-sx2qw\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.435983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.435903 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-certs\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.435983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.435933 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-data\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.436261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436017 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-socket\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.436261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436059 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-podinfo\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.436261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436106 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-envoy\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.436261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436135 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4daf7ff6-9658-4276-90f2-a148d32143af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4daf7ff6-9658-4276-90f2-a148d32143af-istio-podinfo\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.436261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436183 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-certs\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.436261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436226 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-token\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-token\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436281 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-data\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436321 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-podinfo\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436363 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-credential-socket\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436430 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-socket\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436463 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-kube-api-access-m2dr7\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-kube-api-access-m2dr7\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436500 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-credential-socket\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436552 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-certs\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436616 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-podinfo\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436678 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-data\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436713 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-envoy\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436804 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-socket\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436852 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-token\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-token\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436891 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4daf7ff6-9658-4276-90f2-a148d32143af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4daf7ff6-9658-4276-90f2-a148d32143af-istiod-ca-cert\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436971 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-data\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437036 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-kube-api-access-nwtwr\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-kube-api-access-nwtwr\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437089 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-token\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-token\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437117 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-istio-token\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-istio-token\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437167 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-envoy\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437211 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-certs\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437295 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-envoy\") pod \"4daf7ff6-9658-4276-90f2-a148d32143af\" (UID: \"4daf7ff6-9658-4276-90f2-a148d32143af\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437354 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-credential-socket\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437415 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-podinfo\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437462 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istiod-ca-cert\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437501 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884ab263-fded-4dec-b6f5-bb9799fbb140-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884ab263-fded-4dec-b6f5-bb9799fbb140-istiod-ca-cert\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437545 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-credential-socket\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437608 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-socket\") pod \"884ab263-fded-4dec-b6f5-bb9799fbb140\" (UID: \"884ab263-fded-4dec-b6f5-bb9799fbb140\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437643 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-credential-socket\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437685 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-kube-api-access-7dh9j\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-kube-api-access-7dh9j\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437729 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-envoy\") pod \"aa15275f-8808-4e9a-bb73-875dedce34c6\" (UID: \"aa15275f-8808-4e9a-bb73-875dedce34c6\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437770 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-kube-api-access-vlm6f\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-kube-api-access-vlm6f\") pod \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\" (UID: \"0341c0e9-195b-4953-bcf4-e5f34f7949d9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437868 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istiod-ca-cert\") pod \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\" (UID: \"d50172ea-80ec-49ee-8534-f8f9841a4fc9\") " Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.438727 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.438866 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.439035 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.439154 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.439279 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.439580 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5h76\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-kube-api-access-z5h76\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.439677 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.439737 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.440248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.439814 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.445794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.445761 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.447472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.447446 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q"] Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.414928 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa15275f-8808-4e9a-bb73-875dedce34c6-istiod-ca-cert" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436548 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-certs" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.436750 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-data" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437309 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-socket" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.437545 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-certs" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.445633 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-socket" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.446028 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-data" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.446175 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-kube-api-access-sx2qw" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "kube-api-access-sx2qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.451497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.451286 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-certs" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.452022 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.451984 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-data" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452163 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452189 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452235 5610 cpu_manager.go:414] "containerMap: removing container" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452244 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452284 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452291 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452341 5610 cpu_manager.go:414] "containerMap: removing container" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452349 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452374 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452381 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452437 5610 cpu_manager.go:414] "containerMap: removing container" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.452444 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.455861 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-data" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.456732 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.456771 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.456809 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.456825 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.456851 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.456872 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" containerName="istio-proxy" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.457160 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-credential-socket" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.457376 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-socket" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.458785 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-socket" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.459596 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-credential-socket" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.459808 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-certs" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.460323 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-podinfo" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.460594 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-data" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.460909 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4daf7ff6-9658-4276-90f2-a148d32143af-istiod-ca-cert" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.461480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.461270 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istiod-ca-cert" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.462957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.461561 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-certs" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.469373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.469346 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-envoy" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.473616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.470692 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-credential-socket" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.473616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.471044 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884ab263-fded-4dec-b6f5-bb9799fbb140-istiod-ca-cert" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.473616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.471693 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-credential-socket" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.487587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.481318 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.487587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.482034 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.487587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.482283 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.487587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.486173 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istiod-ca-cert" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.487587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.486453 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.487587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.486771 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-socket" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.495790 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-kube-api-access-7dh9j" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "kube-api-access-7dh9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.495862 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-token" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.496790 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-kube-api-access-m2dr7" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "kube-api-access-m2dr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.500163 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q"] Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.500302 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.501885 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-credential-socket" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.504264 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-envoy" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.504353 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-istio-token" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.504414 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-envoy" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.505468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.505333 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.507906 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.507869 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-podinfo" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.508130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.508102 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4daf7ff6-9658-4276-90f2-a148d32143af-istio-podinfo" pod "4daf7ff6-9658-4276-90f2-a148d32143af" (UID: "4daf7ff6-9658-4276-90f2-a148d32143af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.508315 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.508289 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-token" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.508515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.508487 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-token" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.519460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.513202 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.519460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.513252 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh"] Oct 29 22:39:55.519460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.514141 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-podinfo" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.519460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.514230 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-kube-api-access-nwtwr" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "kube-api-access-nwtwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.519460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.514287 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-envoy" pod "aa15275f-8808-4e9a-bb73-875dedce34c6" (UID: "aa15275f-8808-4e9a-bb73-875dedce34c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.519460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.514637 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.519460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.514739 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5h76\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-kube-api-access-z5h76\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.519823 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.519798 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-podinfo" pod "d50172ea-80ec-49ee-8534-f8f9841a4fc9" (UID: "d50172ea-80ec-49ee-8534-f8f9841a4fc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.524866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.522074 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-envoy" pod "884ab263-fded-4dec-b6f5-bb9799fbb140" (UID: "884ab263-fded-4dec-b6f5-bb9799fbb140"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.525019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.522079 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-kube-api-access-vlm6f" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "kube-api-access-vlm6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.525113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.524796 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-token" pod "0341c0e9-195b-4953-bcf4-e5f34f7949d9" (UID: "0341c0e9-195b-4953-bcf4-e5f34f7949d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.527810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.526997 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh"] Oct 29 22:39:55.527810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.527113 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.533893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.533870 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m"] Oct 29 22:39:55.541740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.540698 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istiod-ca-cert\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.541740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.540778 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-certs\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.541740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.541253 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istiod-ca-cert" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:55.541740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.541472 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-certs" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.542056 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.541980 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-socket" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.542128 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-socket\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.542239 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-data\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.542278 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-token\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.542415 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-podinfo\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.542484 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-envoy\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.542529 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-credential-socket\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.542565 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-kube-api-access-zq8zl\") pod \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\" (UID: \"9ae8da64-61ec-463f-b44f-299bd4c6d1b4\") " Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543247 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543269 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543284 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543298 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543314 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/884ab263-fded-4dec-b6f5-bb9799fbb140-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543327 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543340 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543355 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543369 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543382 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dh9j\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-kube-api-access-7dh9j\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543415 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543428 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlm6f\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-kube-api-access-vlm6f\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543442 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543457 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa15275f-8808-4e9a-bb73-875dedce34c6-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543470 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543483 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543496 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543509 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543523 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2qw\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-kube-api-access-sx2qw\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543537 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543553 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543568 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543582 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543596 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543609 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4daf7ff6-9658-4276-90f2-a148d32143af-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543622 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543637 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543651 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543664 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543677 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543693 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543707 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2dr7\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-kube-api-access-m2dr7\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543722 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daf7ff6-9658-4276-90f2-a148d32143af-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543735 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543750 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543763 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50172ea-80ec-49ee-8534-f8f9841a4fc9-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543776 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543789 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0341c0e9-195b-4953-bcf4-e5f34f7949d9-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543804 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543818 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543831 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4daf7ff6-9658-4276-90f2-a148d32143af-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543843 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa15275f-8808-4e9a-bb73-875dedce34c6-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543856 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543870 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtwr\" (UniqueName: \"kubernetes.io/projected/884ab263-fded-4dec-b6f5-bb9799fbb140-kube-api-access-nwtwr\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543885 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0341c0e9-195b-4953-bcf4-e5f34f7949d9-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543898 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4daf7ff6-9658-4276-90f2-a148d32143af-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543910 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.544217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.543924 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/884ab263-fded-4dec-b6f5-bb9799fbb140-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.548097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.545625 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-credential-socket" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.548097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.545680 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-data" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.548097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.547923 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m"] Oct 29 22:39:55.548097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.548033 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.548796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.548762 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-envoy" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:55.553081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.552570 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-token" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.555282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.553629 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f76fb5-openshift-default-dockercfg-lxhl2\"" Oct 29 22:39:55.557340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.556101 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-podinfo" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:55.557340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.556943 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-kube-api-access-zq8zl" pod "9ae8da64-61ec-463f-b44f-299bd4c6d1b4" (UID: "9ae8da64-61ec-463f-b44f-299bd4c6d1b4"). InnerVolumeSpecName "kube-api-access-zq8zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:55.557340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.557167 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:55.561312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.560489 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl"] Oct 29 22:39:55.601570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.601544 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl"] Oct 29 22:39:55.601769 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.601750 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn"] Oct 29 22:39:55.601971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.601955 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.609159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.609121 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.643104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.643076 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e224cd-openshift-default-dockercfg-98zkf\"" Oct 29 22:39:55.643250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.643232 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1ae775-openshift-default-dockercfg-b8gvg\"" Oct 29 22:39:55.652768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.652733 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.652918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.652791 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106bfcd-df18-4806-9286-c5b96aff17d5-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.652918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.652845 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.652918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.652885 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.653087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.652929 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230c618f-fa51-489b-9075-45cb39b854a3-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.653087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.652963 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.653087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.652994 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.653087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653045 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.653087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653073 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.653360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653161 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.653360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653215 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.653360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653240 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.653360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653263 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.653360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653303 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.653655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653417 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdns\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.653655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653455 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.653655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653478 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.653655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653515 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230c618f-fa51-489b-9075-45cb39b854a3-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.653655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653558 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.653655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653603 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8j4q\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-kube-api-access-x8j4q\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.653655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653640 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653666 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653691 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5q79\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-kube-api-access-n5q79\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653723 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653793 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653827 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e57308-cd83-4966-9847-e8851bd7ce38-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653874 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653954 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653969 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653984 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.653999 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8zl\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-kube-api-access-zq8zl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.654016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.654013 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.654586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.654026 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ae8da64-61ec-463f-b44f-299bd4c6d1b4-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:55.668066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.668018 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn"] Oct 29 22:39:55.713125 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.713035 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6"] Oct 29 22:39:55.734280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.734205 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6"] Oct 29 22:39:55.734280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.734253 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b"] Oct 29 22:39:55.734479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.734338 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:55.737076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.737047 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-lst5b"] Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.763796 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c06e05-dockercfg-thz8p\"" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.763889 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c06e05\"" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.766859 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767061 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-credential-socket\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767117 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-token\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767148 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767172 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-podinfo\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767201 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istiod-ca-cert\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767232 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e57308-cd83-4966-9847-e8851bd7ce38-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767280 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767317 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9943a5e-db25-4438-bdf0-74979290b12e-istio-podinfo\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767360 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767387 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbvm\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-kube-api-access-8dbvm\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767441 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-certs\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767468 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106bfcd-df18-4806-9286-c5b96aff17d5-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767531 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-socket\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767582 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-socket\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767670 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-credential-socket\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767715 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767769 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767828 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230c618f-fa51-489b-9075-45cb39b854a3-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767868 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767909 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767968 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.767998 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.768039 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-istio-token\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.768085 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-envoy\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.774067 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.774437 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.775137 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e57308-cd83-4966-9847-e8851bd7ce38-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.775405 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.775731 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.778862 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106bfcd-df18-4806-9286-c5b96aff17d5-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.780240 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.780887 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230c618f-fa51-489b-9075-45cb39b854a3-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.781577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.781344 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.781701 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795017 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-data\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795102 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795165 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795191 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795215 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795262 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795324 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-certs\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795445 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-envoy\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795472 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-data\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795508 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdns\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795547 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795576 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795619 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230c618f-fa51-489b-9075-45cb39b854a3-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795641 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9943a5e-db25-4438-bdf0-74979290b12e-istiod-ca-cert\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795687 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795755 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8j4q\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-kube-api-access-x8j4q\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795788 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9pd\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-kube-api-access-dg9pd\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795815 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795839 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.795860 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5q79\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-kube-api-access-n5q79\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.796666 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.797488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.797322 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.798663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.797562 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.808699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.805429 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.808699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.805959 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.808699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.806151 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230c618f-fa51-489b-9075-45cb39b854a3-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.808699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.807104 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.808699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.807578 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.808699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.807794 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.808699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.808220 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.808699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.808661 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.809687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.809467 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.809687 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.809643 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.823747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.823696 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.824421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.824381 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdns\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.825160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.825117 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5q79\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-kube-api-access-n5q79\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.829580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.829122 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8j4q\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-kube-api-access-x8j4q\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.829580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.829385 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898549 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9pd\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-kube-api-access-dg9pd\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898616 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-credential-socket\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898666 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-token\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898692 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-podinfo\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898715 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istiod-ca-cert\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898768 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9943a5e-db25-4438-bdf0-74979290b12e-istio-podinfo\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898811 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbvm\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-kube-api-access-8dbvm\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898850 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-certs\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898881 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-socket\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898920 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrqq\" (UniqueName: \"kubernetes.io/projected/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-kube-api-access-4xrqq\") pod \"payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6\" (UID: \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898954 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-socket\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.898982 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-credential-socket\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.899034 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6\" (UID: \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.899182 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-istio-token\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.899228 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-envoy\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.899237 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-credential-socket\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.899258 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-data\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.899432 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-certs\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.900166 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istiod-ca-cert\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.900849 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-socket\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.901193 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-credential-socket\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.902494 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-certs\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.902541 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-envoy\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.902569 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-data\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.903249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.902640 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9943a5e-db25-4438-bdf0-74979290b12e-istiod-ca-cert\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.904969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.903372 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9943a5e-db25-4438-bdf0-74979290b12e-istiod-ca-cert\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.904969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.903664 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-data\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.904969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.903893 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-certs\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.904969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.904366 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-data\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.906417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.906373 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-envoy\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.909588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.909540 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-socket\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.910166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.910138 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-podinfo\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.913401 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.913360 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-envoy\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.913848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.913829 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9943a5e-db25-4438-bdf0-74979290b12e-istio-podinfo\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.914667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.914460 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:55.917924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.917859 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-token\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.918636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.918617 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9pd\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-kube-api-access-dg9pd\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.924132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.922296 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:55.924796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.924771 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbvm\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-kube-api-access-8dbvm\") pod \"e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.926899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.926854 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-istio-token\") pod \"e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:55.931478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.931385 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:55.972685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.970899 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:39:55.976662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:55.976142 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:56.000235 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:39:56.000194 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e71b7e3_f1c1_45e2_a76a_1d70917e6b93.slice/crio-60cce4f06833074a3e03ef5bd08f124adc1e6cd76647a0448156f61179da27e7 WatchSource:0}: Error finding container 60cce4f06833074a3e03ef5bd08f124adc1e6cd76647a0448156f61179da27e7: Status 404 returned error can't find the container with id 60cce4f06833074a3e03ef5bd08f124adc1e6cd76647a0448156f61179da27e7 Oct 29 22:39:56.008962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.004301 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms"] Oct 29 22:39:56.008962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.004331 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrqq\" (UniqueName: \"kubernetes.io/projected/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-kube-api-access-4xrqq\") pod \"payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6\" (UID: \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:56.008962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.004384 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6\" (UID: \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:56.008962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.005225 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6\" (UID: \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:56.008962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.006981 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.008962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.007072 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.008962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.007124 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.038179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.037280 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrqq\" (UniqueName: \"kubernetes.io/projected/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-kube-api-access-4xrqq\") pod \"payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6\" (UID: \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:56.044294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.044247 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr"] Oct 29 22:39:56.045842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.045817 5610 cpu_manager.go:414] "containerMap: removing container" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" Oct 29 22:39:56.045842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.045843 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" Oct 29 22:39:56.046084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.046069 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerName="istio-proxy" Oct 29 22:39:56.096995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.096960 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:56.108784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.108740 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-socket\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.110632 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-socket" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.112487 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-envoy\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.112556 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-certs\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.112713 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-credential-socket\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.112831 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-data\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.112905 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-token\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-token\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.112975 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istiod-ca-cert\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.113035 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-kube-api-access-6vh85\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-kube-api-access-6vh85\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.113086 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-podinfo\") pod \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\" (UID: \"27ff1219-04a6-4395-b0e1-2ddc2cfb534a\") " Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.113196 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-credential-socket" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:56.113706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.113423 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-certs" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:56.114680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.114196 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istiod-ca-cert" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:39:56.114680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.114491 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.114680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.114510 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.114680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.114525 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.114680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.114543 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.123515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.123486 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-data" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:56.127296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.127215 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-podinfo" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:39:56.132105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.131837 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr"] Oct 29 22:39:56.132491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.132421 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.135259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.135233 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-envoy" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:39:56.151411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.141350 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-token" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:56.151411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.141445 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-kube-api-access-6vh85" pod "27ff1219-04a6-4395-b0e1-2ddc2cfb534a" (UID: "27ff1219-04a6-4395-b0e1-2ddc2cfb534a"). InnerVolumeSpecName "kube-api-access-6vh85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:39:56.234113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.199149 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q"] Oct 29 22:39:56.234113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.225205 5610 patch_prober.go:29] interesting pod/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:39:56.234113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.225283 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:56.238572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.238547 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.238776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.238723 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.238993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.238970 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdq8g\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-kube-api-access-jdq8g\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.239132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.239120 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.239234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.239217 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.239376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.239365 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.239730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.239712 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.240028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.240001 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.241631 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.241451 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.241731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.241693 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.241797 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.241744 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.244570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.242756 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.244570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.242952 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.244570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.242971 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.244570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.242984 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vh85\" (UniqueName: \"kubernetes.io/projected/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-kube-api-access-6vh85\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.244570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.243006 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.244570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.243022 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ff1219-04a6-4395-b0e1-2ddc2cfb534a-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:39:56.290472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.285849 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.345793 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdq8g\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-kube-api-access-jdq8g\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.345901 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.345937 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.346026 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.346143 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.346176 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.346272 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.346431 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.346492 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.347004 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.347649 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.348265 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.348538 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.349731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.348875 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.370433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.370353 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.372901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.372847 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdq8g\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-kube-api-access-jdq8g\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.385167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.385110 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.386252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.386194 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.389066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.387564 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" event={"ID":"aa15275f-8808-4e9a-bb73-875dedce34c6","Type":"ContainerDied","Data":"bd5239840eb1d16e6b90a0640839e071ab07ce73d067dd3406c01b2102f24e75"} Oct 29 22:39:56.389066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.387632 5610 scope.go:122] "RemoveContainer" containerID="bdd0e9146b6f091fd3efa903052377e1da705dd6981cfa93da504f47803949a4" Oct 29 22:39:56.389066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.387814 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx" Oct 29 22:39:56.396431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.394580 5610 generic.go:356] "Generic (PLEG): container finished" podID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" containerID="4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b" exitCode=0 Oct 29 22:39:56.396431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.394720 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" event={"ID":"27ff1219-04a6-4395-b0e1-2ddc2cfb534a","Type":"ContainerDied","Data":"4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b"} Oct 29 22:39:56.396431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.394754 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" event={"ID":"27ff1219-04a6-4395-b0e1-2ddc2cfb534a","Type":"ContainerDied","Data":"bf5c937af86234b8a3af3f05c003fe5a9db4cc7ea1a5eab6b81eb0b452088d99"} Oct 29 22:39:56.396431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.394828 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9" Oct 29 22:39:56.399151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.398871 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" event={"ID":"9ae8da64-61ec-463f-b44f-299bd4c6d1b4","Type":"ContainerDied","Data":"2696042730eac3f54a2ff1f48e720ba044001f56f791da21f8263a7020cd7354"} Oct 29 22:39:56.399151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.399007 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk" Oct 29 22:39:56.408596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.408510 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" Oct 29 22:39:56.408596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.408548 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg" event={"ID":"0341c0e9-195b-4953-bcf4-e5f34f7949d9","Type":"ContainerDied","Data":"ede8cdec8887fe556d49f10308ac82661dac89710c55b5242aa558b7e4297601"} Oct 29 22:39:56.418572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.418546 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" event={"ID":"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93","Type":"ContainerStarted","Data":"60cce4f06833074a3e03ef5bd08f124adc1e6cd76647a0448156f61179da27e7"} Oct 29 22:39:56.423002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.422946 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" event={"ID":"d50172ea-80ec-49ee-8534-f8f9841a4fc9","Type":"ContainerDied","Data":"c122f7139515deea38f89ebed7439aa05a6bba0b7e48966f42e7aecc23c8e4ae"} Oct 29 22:39:56.423563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.423266 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk" Oct 29 22:39:56.428700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.428587 5610 scope.go:122] "RemoveContainer" containerID="4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b" Oct 29 22:39:56.431546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.431517 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" event={"ID":"4daf7ff6-9658-4276-90f2-a148d32143af","Type":"ContainerDied","Data":"b1c7c6c5c88ed3e7a095cfb25c276a9af0840a7fe8e9f46ee8637c27fd3113c2"} Oct 29 22:39:56.431844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.431823 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc" Oct 29 22:39:56.435212 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.435156 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" event={"ID":"884ab263-fded-4dec-b6f5-bb9799fbb140","Type":"ContainerDied","Data":"dfee91db9eb2f17be8d4e9137e5e56085e1849aa113055600ea899e98aec4977"} Oct 29 22:39:56.435314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.435277 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97" Oct 29 22:39:56.445680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.442723 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" event={"ID":"230c618f-fa51-489b-9075-45cb39b854a3","Type":"ContainerStarted","Data":"d07c6bc182c4028e5df7e5cf0d4d285cd1179f46f6c9fd7b9bab2181c87f0d0c"} Oct 29 22:39:56.465128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.465092 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx"] Oct 29 22:39:56.475964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.475943 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-wnvlx"] Oct 29 22:39:56.515619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.515499 5610 scope.go:122] "RemoveContainer" containerID="4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b" Oct 29 22:39:56.517215 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:39:56.517026 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b\": container with ID starting with 4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b not found: ID does not exist" containerID="4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b" Oct 29 22:39:56.517215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.517070 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b"} err="failed to get container status \"4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b\": rpc error: code = NotFound desc = could not find container \"4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b\": container with ID starting with 4c121dc6bc0a62a75e8cf409dde17ea3f2bab93fc80fa72ea8b11006f86b0f6b not found: ID does not exist" Oct 29 22:39:56.517215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.517093 5610 scope.go:122] "RemoveContainer" containerID="d01c5755e5f00609f395119461e1ba37dd2d76c715c2d27a2ec7524b19db22ca" Oct 29 22:39:56.520236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.519569 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh"] Oct 29 22:39:56.530673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.525703 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.530673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.525793 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.530673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.525839 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.547952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.547896 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk"] Oct 29 22:39:56.555201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.555170 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:56.563450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.563293 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-c8dxk"] Oct 29 22:39:56.583207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.582053 5610 scope.go:122] "RemoveContainer" containerID="52d035c53772f65f13487294cb042e7097a061bc150199f45157d10a2d2455e4" Oct 29 22:39:56.590024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.588663 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m"] Oct 29 22:39:56.597645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.597050 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.597645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.597135 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.597645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.597179 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.630463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.626140 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.630463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.626228 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.630463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.626270 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.630463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.626913 5610 scope.go:122] "RemoveContainer" containerID="46bd259206ee3467dc4e64e7c803b98db1c684fb69456dbf88c739204d0f9d20" Oct 29 22:39:56.630463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.628903 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl"] Oct 29 22:39:56.636473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.633601 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk"] Oct 29 22:39:56.639602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.639574 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-r25nk"] Oct 29 22:39:56.724673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.701510 5610 scope.go:122] "RemoveContainer" containerID="16db3dc1ec23bd55692695596d10f64a286cb3db6deb7dd965bcb878dd76f736" Oct 29 22:39:56.730806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.730746 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc"] Oct 29 22:39:56.735520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.734869 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.736638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.736610 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.736814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.736794 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:56.744674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.744619 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8d3632-openshift-default-5ddf4457d9-xbqjc"] Oct 29 22:39:56.761769 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.760440 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn"] Oct 29 22:39:56.829432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.829373 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97"] Oct 29 22:39:56.850675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.850373 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-bsp97"] Oct 29 22:39:56.850675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.850444 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9"] Oct 29 22:39:56.857153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.855385 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-s2wp9"] Oct 29 22:39:56.860586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.860205 5610 scope.go:122] "RemoveContainer" containerID="c97f5a6f53159efd9f0210794f0704b4bd7a27fe5aa9e5293d8bfd2af8ac2c9b" Oct 29 22:39:56.862445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.862425 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg"] Oct 29 22:39:56.872057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:56.872024 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4a0287-openshift-default-6b4449b6b5-xzzpg"] Oct 29 22:39:57.034557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.034429 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6"] Oct 29 22:39:57.104633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.104037 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr"] Oct 29 22:39:57.142288 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.142240 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:57.142613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.142581 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:57.142830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.142800 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:39:57.476690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.476653 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" event={"ID":"1106bfcd-df18-4806-9286-c5b96aff17d5","Type":"ContainerStarted","Data":"97ed0bd2e794f6cda1c613130eead2d3b57ede2a38487af99d86bfe7c6d93b94"} Oct 29 22:39:57.476994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.476948 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" event={"ID":"1106bfcd-df18-4806-9286-c5b96aff17d5","Type":"ContainerStarted","Data":"949f7a907c0718ab39439a797bbe2927b2ca38f9ed8dedeadc08462a20afbb4b"} Oct 29 22:39:57.512503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.499518 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" event={"ID":"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf","Type":"ContainerStarted","Data":"cf2da527c5885205f129e651c20989d67affc26de6ef1753a5589413e27a5acb"} Oct 29 22:39:57.512503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.503069 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" event={"ID":"d3e57308-cd83-4966-9847-e8851bd7ce38","Type":"ContainerStarted","Data":"1e08596386a34a03fb9c83c21c8992be14b5171e309b986d7042c0ba533f1303"} Oct 29 22:39:57.512503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.503123 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" event={"ID":"d3e57308-cd83-4966-9847-e8851bd7ce38","Type":"ContainerStarted","Data":"e98b16876450799c31dcae39a67fe281e7c1501b96c7a7ed9fc93f9c9729b30c"} Oct 29 22:39:57.519570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.518575 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" event={"ID":"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93","Type":"ContainerStarted","Data":"d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811"} Oct 29 22:39:57.522187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.522160 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" event={"ID":"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36","Type":"ContainerStarted","Data":"157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e"} Oct 29 22:39:57.522456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.522374 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" event={"ID":"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36","Type":"ContainerStarted","Data":"3c1136173f78fd8c09ac8770cd8baf483518813b0f6c159d8fed03a68a3b34a8"} Oct 29 22:39:57.529592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.528801 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podStartSLOduration=6.528783531 podStartE2EDuration="6.528783531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:57.527163648 +0000 UTC m=+1460.008481142" watchObservedRunningTime="2026-10-29 22:39:57.528783531 +0000 UTC m=+1460.010101024" Oct 29 22:39:57.535700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.535672 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" event={"ID":"e5186bd9-b725-4a0d-b5a7-e8adefc36228","Type":"ContainerStarted","Data":"91395f05b1a0032600d5fa46ed0ebea712d67b19581aad27fb4452ee7a714b8d"} Oct 29 22:39:57.540521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.540495 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" event={"ID":"a9943a5e-db25-4438-bdf0-74979290b12e","Type":"ContainerStarted","Data":"772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec"} Oct 29 22:39:57.541013 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.540989 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" event={"ID":"a9943a5e-db25-4438-bdf0-74979290b12e","Type":"ContainerStarted","Data":"0e02333b94e416f31cd4ecaf7649f9e352ab6c507d86d57de3b681b1ebf05bdf"} Oct 29 22:39:57.542761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.542693 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" event={"ID":"230c618f-fa51-489b-9075-45cb39b854a3","Type":"ContainerStarted","Data":"d2eea1347eb49afaa4a201b940cd8b2c34129f4502956ddfb63131d725e9204d"} Oct 29 22:39:57.560688 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.560281 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:39:57.560688 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.560382 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:39:57.560688 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.560440 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:39:57.637710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.632803 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podStartSLOduration=12.632784647 podStartE2EDuration="12.632784647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:57.630834984 +0000 UTC m=+1460.112152478" watchObservedRunningTime="2026-10-29 22:39:57.632784647 +0000 UTC m=+1460.114102144" Oct 29 22:39:57.684053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.683979 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podStartSLOduration=8.683959415 podStartE2EDuration="8.683959415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:57.584949091 +0000 UTC m=+1460.066266596" watchObservedRunningTime="2026-10-29 22:39:57.683959415 +0000 UTC m=+1460.165276911" Oct 29 22:39:57.691334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.691294 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0341c0e9-195b-4953-bcf4-e5f34f7949d9" path="/var/lib/kubelet/pods/0341c0e9-195b-4953-bcf4-e5f34f7949d9/volumes" Oct 29 22:39:57.693298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.693262 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ff1219-04a6-4395-b0e1-2ddc2cfb534a" path="/var/lib/kubelet/pods/27ff1219-04a6-4395-b0e1-2ddc2cfb534a/volumes" Oct 29 22:39:57.694516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.694486 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4daf7ff6-9658-4276-90f2-a148d32143af" path="/var/lib/kubelet/pods/4daf7ff6-9658-4276-90f2-a148d32143af/volumes" Oct 29 22:39:57.695748 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.695721 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884ab263-fded-4dec-b6f5-bb9799fbb140" path="/var/lib/kubelet/pods/884ab263-fded-4dec-b6f5-bb9799fbb140/volumes" Oct 29 22:39:57.696529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.696503 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae8da64-61ec-463f-b44f-299bd4c6d1b4" path="/var/lib/kubelet/pods/9ae8da64-61ec-463f-b44f-299bd4c6d1b4/volumes" Oct 29 22:39:57.697311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.697272 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa15275f-8808-4e9a-bb73-875dedce34c6" path="/var/lib/kubelet/pods/aa15275f-8808-4e9a-bb73-875dedce34c6/volumes" Oct 29 22:39:57.756832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.756797 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50172ea-80ec-49ee-8534-f8f9841a4fc9" path="/var/lib/kubelet/pods/d50172ea-80ec-49ee-8534-f8f9841a4fc9/volumes" Oct 29 22:39:57.780616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.757831 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7084709-6ae5-4bba-b2b1-6c590b3ee21d" path="/var/lib/kubelet/pods/d7084709-6ae5-4bba-b2b1-6c590b3ee21d/volumes" Oct 29 22:39:57.867498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.858063 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:39:57.881472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.832083 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:39:57.881759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.881727 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:39:57.882136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.691663 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podStartSLOduration=6.691648706 podStartE2EDuration="6.691648706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:57.683166814 +0000 UTC m=+1460.164484310" watchObservedRunningTime="2026-10-29 22:39:57.691648706 +0000 UTC m=+1460.172966196" Oct 29 22:39:57.929342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.929279 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:39:57.931283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.931260 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:39:57.932862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.932845 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:57.941481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.935184 5610 patch_prober.go:29] interesting pod/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:39:57.941711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.941687 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 29 22:39:57.942023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.935229 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:39:57.942154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.942131 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:39:57.957842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.957810 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:39:57.958067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.958037 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:39:57.980332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:57.980301 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:58.020232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.003805 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" podStartSLOduration=12.003784626 podStartE2EDuration="12.003784626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:57.910447702 +0000 UTC m=+1460.391765217" watchObservedRunningTime="2026-10-29 22:39:58.003784626 +0000 UTC m=+1460.485102121" Oct 29 22:39:58.164473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.138097 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podStartSLOduration=9.138074804 podStartE2EDuration="9.138074804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:57.955768707 +0000 UTC m=+1460.437086201" watchObservedRunningTime="2026-10-29 22:39:58.138074804 +0000 UTC m=+1460.619392298" Oct 29 22:39:58.568422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.562871 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:39:58.568422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.562947 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:39:58.613744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.612750 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" event={"ID":"e5186bd9-b725-4a0d-b5a7-e8adefc36228","Type":"ContainerStarted","Data":"dc5c4c7f97ce7a1608be64e9f0b1c9ccac889a27fc2b83f439e97d1d64bb2e6e"} Oct 29 22:39:58.696424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.693116 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podStartSLOduration=52.693095312 podStartE2EDuration="52.693095312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:58.690758837 +0000 UTC m=+1461.172076343" watchObservedRunningTime="2026-10-29 22:39:58.693095312 +0000 UTC m=+1461.174412807" Oct 29 22:39:58.769417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.745222 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" event={"ID":"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf","Type":"ContainerStarted","Data":"e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818"} Oct 29 22:39:58.772645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.772618 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:39:58.772847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.772509 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:39:58.838571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.836910 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" podStartSLOduration=10.836889735 podStartE2EDuration="10.836889735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:39:58.802539436 +0000 UTC m=+1461.283856931" watchObservedRunningTime="2026-10-29 22:39:58.836889735 +0000 UTC m=+1461.318207229" Oct 29 22:39:58.839254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.839223 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:39:58.840160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.840131 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:39:58.931484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.927723 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:39:58.931484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.927780 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:39:58.936433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.932482 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:39:58.936433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.932537 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:39:58.971217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:58.971182 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:59.150467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.149779 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:59.585843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.585810 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:39:59.616128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.616093 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:39:59.616376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.616345 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:39:59.616851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.616830 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:39:59.640419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.617002 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:39:59.844481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.844280 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:39:59.845418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.845217 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:39:59.845418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.845258 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:39:59.879936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.879896 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n"] Oct 29 22:39:59.940437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.937755 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:59.940437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.937816 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:39:59.940437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.938949 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:39:59.940437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.939003 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:39:59.941572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.941478 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:39:59.941572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.941521 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:39:59.948304 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.948280 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:39:59.949029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:39:59.949012 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:40:00.077596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.077561 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:00.809454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.808188 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:40:00.900052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.900006 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:40:00.966384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.939447 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:00.966384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.939536 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:00.966384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.965446 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:00.966384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.965833 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:00.997917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.967743 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:00.997917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.968491 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:00.997917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.997706 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:00.997917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:00.997771 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:01.209522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.161582 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:40:01.209522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.162986 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 29 22:40:01.209522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.163145 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:40:01.209522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.195210 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:01.209522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.195311 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:01.209522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.195464 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:01.212519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.212457 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:01.230447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.213155 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:40:01.347773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.347677 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:01.633475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.579272 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:01.633475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.579329 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:01.723449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.715545 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:01.746621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.741603 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:01.858216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.858162 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 29 22:40:01.886069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.885980 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:40:01.903461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.889290 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:01.932201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.932163 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:01.957420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.914808 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:01.959270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.929680 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:01.959270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.959179 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:01.970356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.969100 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:01.970356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.969153 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:02.026416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:01.990823 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:02.222476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.173049 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:02.347419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.323117 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:40:02.347419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.323210 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:40:02.347419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.323270 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:40:02.729472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.710804 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:02.766059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.766019 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:02.890611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.880672 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:02.977525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.968758 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:02.977525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.968847 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:02.999484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.997504 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:02.999484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:02.997569 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:03.022896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.022848 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:40:03.031431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.023471 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:40:03.032021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.031989 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:03.330830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.330745 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:03.740627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.351934 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:03.763429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.741096 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:03.861289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.534295 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:40:03.915533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.541604 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w"] Oct 29 22:40:03.919532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.439618 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" podUID="10fcae09-de89-49fc-98f1-3bdaf39751b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 29 22:40:03.920634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.378023 5610 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-vn7xs container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:40:03.920801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.920776 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" podUID="d0c9a67d-21dc-44cf-ab1f-1ef20fba70af" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.12:9443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:03.920946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.571540 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:03.945083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.945036 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:03.945335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.795600 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:03.945503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.945479 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:03.946517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.835462 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:03.946517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.946199 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:03.946517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.879264 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:03.946517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.946247 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:03.946517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.919431 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:03.946517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.946283 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:03.947065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.946975 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:03.947065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:03.947021 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:04.009268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.009178 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:04.046129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.040282 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:40:04.046410 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.046363 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:04.130139 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.114807 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1ae775\"" Oct 29 22:40:04.208449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.199136 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:04.216521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.212542 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:40:04.412491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.412377 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w"] Oct 29 22:40:04.443087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.442238 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:04.529307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.529268 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459197a3-936b-4f21-9872-521530fc1247-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w\" (UID: \"459197a3-936b-4f21-9872-521530fc1247\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:04.529873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.529842 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/459197a3-936b-4f21-9872-521530fc1247-kube-api-access-sbs8v\") pod \"payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w\" (UID: \"459197a3-936b-4f21-9872-521530fc1247\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:04.573663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.573631 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:04.581493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.581454 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:04.581703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.577039 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:04.581818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.581798 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:04.636281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.636234 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459197a3-936b-4f21-9872-521530fc1247-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w\" (UID: \"459197a3-936b-4f21-9872-521530fc1247\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:04.636873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.636846 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/459197a3-936b-4f21-9872-521530fc1247-kube-api-access-sbs8v\") pod \"payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w\" (UID: \"459197a3-936b-4f21-9872-521530fc1247\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:04.688817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.688718 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459197a3-936b-4f21-9872-521530fc1247-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w\" (UID: \"459197a3-936b-4f21-9872-521530fc1247\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:04.826664 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.826630 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/459197a3-936b-4f21-9872-521530fc1247-kube-api-access-sbs8v\") pod \"payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w\" (UID: \"459197a3-936b-4f21-9872-521530fc1247\") " pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:04.894518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.894465 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" podUID="73580531-7db0-4a6a-9f0d-14ac98b530bd" containerName="payload-processing" containerID="cri-o://6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51" gracePeriod=25 Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.894762 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.978737 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.978796 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.978882 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.978904 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.979516 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:04.979555 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.037651 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:05.045638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.045514 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:05.088280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.061628 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:05.089598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.078049 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:05.271541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.271477 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:05.320500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.307653 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:05.565423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.565369 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:05.596474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.596371 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:05.597093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.596701 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:05.597635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.597596 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:05.597798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.597777 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:05.956548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.956498 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:05.962340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.962307 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:05.962598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.962570 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:05.991636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.990669 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:05.991636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:05.990730 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:06.016167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.015503 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:06.016167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.015558 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:06.072259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.072213 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:40:06.184002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.183962 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:06.266529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.266440 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq"] Oct 29 22:40:06.266959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.266929 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" containerID="cri-o://fdb70db627efec49f6fd61bc2c108f8c087dc3239fed979801b179a640efbf1e" gracePeriod=30 Oct 29 22:40:06.336413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.294438 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:06.415806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.367532 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:06.415806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.381511 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2"] Oct 29 22:40:06.415806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.381752 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" containerID="cri-o://3c94d9c03e2b7ca03f6461a9c9218d477707f829c7ff264a8ea98db918401ebe" gracePeriod=30 Oct 29 22:40:06.591465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.572324 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:06.591465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.572410 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:06.700701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.619444 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:06.700701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.619515 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:06.899477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.886068 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:06.899477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.886134 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:06.920424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:06.917133 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47"] Oct 29 22:40:07.068418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.043623 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" containerID="cri-o://f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3" gracePeriod=30 Oct 29 22:40:07.358629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.052089 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:07.358629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.052835 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:07.358629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.348171 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:07.358629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.203899 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:07.428449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.116892 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:07.481419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.131410 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:07.592449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.552120 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:07.752506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.661703 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:07.752506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.661842 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:07.982419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:07.982311 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:40:08.058594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.041993 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:08.058594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.042060 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:08.058594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.042461 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:08.058594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.042506 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:08.058594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.052176 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:08.058594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.052238 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:08.058594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.052346 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:08.058594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.052371 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:08.364629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.364519 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:08.457665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.449759 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:08.499040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.498885 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:08.569474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.500514 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:40:08.647475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.629547 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:08.679442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.675503 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:08.679442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.649577 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:08.679442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.675609 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:08.948977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.928837 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:08.948977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.928916 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:08.948977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.929019 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:08.948977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:08.929041 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:09.000834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.000794 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:09.025497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.025450 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:09.033418 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:09.029361 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.341s" Oct 29 22:40:09.150778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.150747 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2"] Oct 29 22:40:09.154529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.154504 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9005-bbf3c9"] Oct 29 22:40:09.170504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.051661 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:40:09.171056 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.162543 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" containerID="cri-o://6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29" gracePeriod=30 Oct 29 22:40:09.204761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.204672 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-bbf3c9"] Oct 29 22:40:09.204914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.204794 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:09.249178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.247770 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwnv\" (UniqueName: \"kubernetes.io/projected/446d1b8e-11af-4aef-b589-a48a14d842b6-kube-api-access-xlwnv\") pod \"test-curl-9005-bbf3c9\" (UID: \"446d1b8e-11af-4aef-b589-a48a14d842b6\") " pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:09.274557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.253437 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c5k5w\"" Oct 29 22:40:09.391420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.387383 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwnv\" (UniqueName: \"kubernetes.io/projected/446d1b8e-11af-4aef-b589-a48a14d842b6-kube-api-access-xlwnv\") pod \"test-curl-9005-bbf3c9\" (UID: \"446d1b8e-11af-4aef-b589-a48a14d842b6\") " pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:09.391420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.390523 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:09.461654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.452847 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:09.545416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.520372 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:09.547091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.529359 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:40:09.547851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.547830 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:40:09.551372 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.551310 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwnv\" (UniqueName: \"kubernetes.io/projected/446d1b8e-11af-4aef-b589-a48a14d842b6-kube-api-access-xlwnv\") pod \"test-curl-9005-bbf3c9\" (UID: \"446d1b8e-11af-4aef-b589-a48a14d842b6\") " pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:09.563786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.563755 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:09.563989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.563963 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:09.572504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.565208 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:09.572751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.572720 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:09.588417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.583700 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w"] Oct 29 22:40:09.815024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.776912 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:09.916531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.916496 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:40:09.979599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.944567 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:40:09.979599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.946796 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:09.979599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.946842 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:09.979599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.946915 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:09.979599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:09.946935 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:10.282418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.266502 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:10.339427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.330895 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" event={"ID":"459197a3-936b-4f21-9872-521530fc1247","Type":"ContainerStarted","Data":"5f7960f0bf2f0f0b71813b9b2fbc20186b3bcb9709be38f1dd1bb5facc9277bb"} Oct 29 22:40:10.548627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.547372 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:40:10.564422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.560595 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:40:10.569527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.569497 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 29 22:40:10.569782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.569754 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 29 22:40:10.605474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.604949 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 29 22:40:10.605474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.605025 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 29 22:40:10.866982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.866952 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:40:10.937450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.937411 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:40:10.939696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.938897 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:40:10.957434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.957408 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:40:10.984414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.957244 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 29 22:40:10.984414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.958476 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 29 22:40:10.984414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.957285 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 29 22:40:10.984414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:10.958777 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 29 22:40:11.309461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.305631 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:40:11.334043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.317639 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 29 22:40:11.451951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.451911 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-bbf3c9"] Oct 29 22:40:11.454264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.454237 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:40:11.482936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.481758 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:40:11.550951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.549504 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:11.562060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.559209 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:40:11.575068 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.574687 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:11.600422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.591158 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:40:11.632808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.613448 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:11.655575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.655540 5610 generic.go:356] "Generic (PLEG): container finished" podID="3017d7fa-514b-426a-b489-a34942e99745" containerID="fdb70db627efec49f6fd61bc2c108f8c087dc3239fed979801b179a640efbf1e" exitCode=0 Oct 29 22:40:11.729369 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.727622 5610 generic.go:356] "Generic (PLEG): container finished" podID="1f583580-6274-4723-8d24-931c393a76c4" containerID="3c94d9c03e2b7ca03f6461a9c9218d477707f829c7ff264a8ea98db918401ebe" exitCode=0 Oct 29 22:40:11.745890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.745866 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:40:11.746337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.746320 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:11.746540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.746520 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-bbf3c9" event={"ID":"446d1b8e-11af-4aef-b589-a48a14d842b6","Type":"ContainerStarted","Data":"5a6373bc7a8733d545a3ba73ab5ab15b9ae5ba427fc37c2804892b8e07bc5a1c"} Oct 29 22:40:11.746846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.746830 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:40:11.747001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.746975 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" event={"ID":"3017d7fa-514b-426a-b489-a34942e99745","Type":"ContainerDied","Data":"fdb70db627efec49f6fd61bc2c108f8c087dc3239fed979801b179a640efbf1e"} Oct 29 22:40:11.747147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.747135 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:40:11.747263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.747251 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" event={"ID":"459197a3-936b-4f21-9872-521530fc1247","Type":"ContainerStarted","Data":"b77d0a8891d0b13d5449bdc3aac6267f19444f70ad3b6bf63fc93690f9debd06"} Oct 29 22:40:11.747366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.747352 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" event={"ID":"1f583580-6274-4723-8d24-931c393a76c4","Type":"ContainerDied","Data":"3c94d9c03e2b7ca03f6461a9c9218d477707f829c7ff264a8ea98db918401ebe"} Oct 29 22:40:11.753208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.753167 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" podStartSLOduration=9.753155826 podStartE2EDuration="9.753155826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:11.741845294 +0000 UTC m=+1474.223162788" watchObservedRunningTime="2026-10-29 22:40:11.753155826 +0000 UTC m=+1474.234473320" Oct 29 22:40:11.815846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.814458 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:40:11.948887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:11.947745 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:40:12.056427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.034344 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:40:12.056427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.036522 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:40:12.349417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.349361 5610 patch_prober.go:29] interesting pod/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:40:12.349583 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.349449 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:40:12.364451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.357352 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:40:12.364451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.361621 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:40:12.456890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.456799 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:40:12.520280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.520249 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr"] Oct 29 22:40:12.521512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.521457 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" Oct 29 22:40:12.521512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.521480 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" Oct 29 22:40:12.521774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.521759 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f583580-6274-4723-8d24-931c393a76c4" containerName="istio-proxy" Oct 29 22:40:12.526527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.526502 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.529625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.529604 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:40:12.545236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545200 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-envoy\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.545361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545313 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-istio-token\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-istio-token\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.545465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545360 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-certs\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.545465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545385 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-kube-api-access-qrqg5\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-kube-api-access-qrqg5\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.545586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545539 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3017d7fa-514b-426a-b489-a34942e99745-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017d7fa-514b-426a-b489-a34942e99745-istio-podinfo\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.545638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545590 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-envoy\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.545638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545619 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-data\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.545734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545654 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-data\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.545734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545679 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-credential-socket\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.545882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545763 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-kube-api-access-sz6xc\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-kube-api-access-sz6xc\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.545882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545845 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f583580-6274-4723-8d24-931c393a76c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f583580-6274-4723-8d24-931c393a76c4-istio-podinfo\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.545992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545886 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-credential-socket\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.545992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545931 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-istio-token\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-istio-token\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.545992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545967 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f583580-6274-4723-8d24-931c393a76c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f583580-6274-4723-8d24-931c393a76c4-istiod-ca-cert\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.546141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.545996 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-socket\") pod \"1f583580-6274-4723-8d24-931c393a76c4\" (UID: \"1f583580-6274-4723-8d24-931c393a76c4\") " Oct 29 22:40:12.546141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546037 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-certs\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.546141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546083 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3017d7fa-514b-426a-b489-a34942e99745-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017d7fa-514b-426a-b489-a34942e99745-istiod-ca-cert\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.546285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546143 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-socket\") pod \"3017d7fa-514b-426a-b489-a34942e99745\" (UID: \"3017d7fa-514b-426a-b489-a34942e99745\") " Oct 29 22:40:12.546485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546461 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr"] Oct 29 22:40:12.546652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546627 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.547231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546782 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf747\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-kube-api-access-bf747\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.547231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546876 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.547231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546903 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.547231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546955 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.547231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.546954 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-certs" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.547231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.547032 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.547231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.547063 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-credential-socket" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.547891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.547852 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-data" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.549253 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.548491 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-credential-socket" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.549253 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.548939 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-socket" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.549816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.549424 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-certs" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.549816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.549474 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f583580-6274-4723-8d24-931c393a76c4-istiod-ca-cert" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:12.549816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.549664 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-socket" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.550048 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550027 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.550334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550285 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.550433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550419 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.550663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550636 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.550663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550660 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.550809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550676 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.550809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550712 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f583580-6274-4723-8d24-931c393a76c4-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.550809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550727 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.550809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550740 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.550809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550753 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.550809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.550787 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.551517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.551094 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-data" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.551517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.551307 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3017d7fa-514b-426a-b489-a34942e99745-istiod-ca-cert" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:12.553952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.553918 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f583580-6274-4723-8d24-931c393a76c4-istio-podinfo" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:12.554464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.554409 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3017d7fa-514b-426a-b489-a34942e99745-istio-podinfo" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:12.555081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.555055 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-istio-token" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:12.557051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.557027 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-kube-api-access-qrqg5" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "kube-api-access-qrqg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:12.557252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.557235 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-envoy" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.557360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.557339 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-kube-api-access-sz6xc" pod "1f583580-6274-4723-8d24-931c393a76c4" (UID: "1f583580-6274-4723-8d24-931c393a76c4"). InnerVolumeSpecName "kube-api-access-sz6xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:12.557507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.557379 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-envoy" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:12.557569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.557508 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-istio-token" pod "3017d7fa-514b-426a-b489-a34942e99745" (UID: "3017d7fa-514b-426a-b489-a34942e99745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:12.650631 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.650593 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf"] Oct 29 22:40:12.651983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.651956 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652006 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652086 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652188 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf747\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-kube-api-access-bf747\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652202 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" Oct 29 22:40:12.652256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652222 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" Oct 29 22:40:12.652256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652253 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652279 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652320 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652433 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652479 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3017d7fa-514b-426a-b489-a34942e99745" containerName="istio-proxy" Oct 29 22:40:12.652731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652601 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652634 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.652987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.652966 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.653484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.653457 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.653670 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.653898 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.653936 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654053 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017d7fa-514b-426a-b489-a34942e99745-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654078 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f583580-6274-4723-8d24-931c393a76c4-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654091 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654105 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqg5\" (UniqueName: \"kubernetes.io/projected/3017d7fa-514b-426a-b489-a34942e99745-kube-api-access-qrqg5\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654121 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017d7fa-514b-426a-b489-a34942e99745-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654136 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654151 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017d7fa-514b-426a-b489-a34942e99745-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654168 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz6xc\" (UniqueName: \"kubernetes.io/projected/1f583580-6274-4723-8d24-931c393a76c4-kube-api-access-sz6xc\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654182 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f583580-6274-4723-8d24-931c393a76c4-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:12.654988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.654968 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.656995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.656971 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.659144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.659120 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv"] Oct 29 22:40:12.659265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.659246 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:12.677042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.673273 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf"] Oct 29 22:40:12.677042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.673380 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:12.677042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.673642 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e224cd\"" Oct 29 22:40:12.677042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.674701 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e224cd-dockercfg-jbttp\"" Oct 29 22:40:12.681276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.681251 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv"] Oct 29 22:40:12.682480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.682343 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf747\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-kube-api-access-bf747\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.684911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.684869 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.759121 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.759081 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6bjd\" (UniqueName: \"kubernetes.io/projected/209974ec-e547-4d58-884b-1d13784bdabf-kube-api-access-h6bjd\") pod \"payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv\" (UID: \"209974ec-e547-4d58-884b-1d13784bdabf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:12.759366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.759341 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t26d\" (UniqueName: \"kubernetes.io/projected/171987f3-1f45-4938-a66f-702a5b025b7b-kube-api-access-7t26d\") pod \"payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf\" (UID: \"171987f3-1f45-4938-a66f-702a5b025b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:12.759481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.759439 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171987f3-1f45-4938-a66f-702a5b025b7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf\" (UID: \"171987f3-1f45-4938-a66f-702a5b025b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:12.759698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.759599 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209974ec-e547-4d58-884b-1d13784bdabf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv\" (UID: \"209974ec-e547-4d58-884b-1d13784bdabf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:12.771191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.771148 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-bbf3c9" event={"ID":"446d1b8e-11af-4aef-b589-a48a14d842b6","Type":"ContainerStarted","Data":"77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453"} Oct 29 22:40:12.774700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.774668 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" event={"ID":"3017d7fa-514b-426a-b489-a34942e99745","Type":"ContainerDied","Data":"8e741f5ac1416e24ed5cc9400ecba758d2142e51dd0d180fe1dd1cdef3cf40c5"} Oct 29 22:40:12.774809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.774739 5610 scope.go:122] "RemoveContainer" containerID="fdb70db627efec49f6fd61bc2c108f8c087dc3239fed979801b179a640efbf1e" Oct 29 22:40:12.775646 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.775609 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq" Oct 29 22:40:12.779135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.779116 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" Oct 29 22:40:12.779595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.779571 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2" event={"ID":"1f583580-6274-4723-8d24-931c393a76c4","Type":"ContainerDied","Data":"b8b5d8886231dac756cab6da92efe5f7daa0391155bd0cd4a26eb77b28287537"} Oct 29 22:40:12.785181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.785127 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:40:12.796731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.796710 5610 scope.go:122] "RemoveContainer" containerID="3c94d9c03e2b7ca03f6461a9c9218d477707f829c7ff264a8ea98db918401ebe" Oct 29 22:40:12.812864 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.812819 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9005-bbf3c9" podStartSLOduration=6.812805454 podStartE2EDuration="6.812805454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:12.810108702 +0000 UTC m=+1475.291426192" watchObservedRunningTime="2026-10-29 22:40:12.812805454 +0000 UTC m=+1475.294122947" Oct 29 22:40:12.848628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.848605 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:12.862163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.861542 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171987f3-1f45-4938-a66f-702a5b025b7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf\" (UID: \"171987f3-1f45-4938-a66f-702a5b025b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:12.862163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.861759 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209974ec-e547-4d58-884b-1d13784bdabf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv\" (UID: \"209974ec-e547-4d58-884b-1d13784bdabf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:12.862163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.862071 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bjd\" (UniqueName: \"kubernetes.io/projected/209974ec-e547-4d58-884b-1d13784bdabf-kube-api-access-h6bjd\") pod \"payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv\" (UID: \"209974ec-e547-4d58-884b-1d13784bdabf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:12.863176 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.863116 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t26d\" (UniqueName: \"kubernetes.io/projected/171987f3-1f45-4938-a66f-702a5b025b7b-kube-api-access-7t26d\") pod \"payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf\" (UID: \"171987f3-1f45-4938-a66f-702a5b025b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:12.869722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.867790 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171987f3-1f45-4938-a66f-702a5b025b7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf\" (UID: \"171987f3-1f45-4938-a66f-702a5b025b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:12.869722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.869289 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209974ec-e547-4d58-884b-1d13784bdabf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv\" (UID: \"209974ec-e547-4d58-884b-1d13784bdabf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:12.909435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.908909 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bjd\" (UniqueName: \"kubernetes.io/projected/209974ec-e547-4d58-884b-1d13784bdabf-kube-api-access-h6bjd\") pod \"payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv\" (UID: \"209974ec-e547-4d58-884b-1d13784bdabf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:12.911270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.911059 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t26d\" (UniqueName: \"kubernetes.io/projected/171987f3-1f45-4938-a66f-702a5b025b7b-kube-api-access-7t26d\") pod \"payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf\" (UID: \"171987f3-1f45-4938-a66f-702a5b025b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:12.958733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.958666 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq"] Oct 29 22:40:12.995811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:12.993933 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-l7wtq"] Oct 29 22:40:13.000968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.000942 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:13.014788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.014695 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:13.039625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.031023 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2"] Oct 29 22:40:13.039625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.038043 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-m2xm2"] Oct 29 22:40:13.134565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.134533 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.189341 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-kube-api-access-pcpwn\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-kube-api-access-pcpwn\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.189840 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-data\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.190968 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-data" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.191043 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-credential-socket\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.191252 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-credential-socket" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.191374 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-envoy\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.191761 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-istio-token\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-istio-token\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.192184 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/949978bd-024c-4b88-849f-e89bd7f191d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949978bd-024c-4b88-849f-e89bd7f191d8-istio-podinfo\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.192277 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-socket\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.192381 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-certs\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.192449 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949978bd-024c-4b88-849f-e89bd7f191d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949978bd-024c-4b88-849f-e89bd7f191d8-istiod-ca-cert\") pod \"949978bd-024c-4b88-849f-e89bd7f191d8\" (UID: \"949978bd-024c-4b88-849f-e89bd7f191d8\") " Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.192749 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj"] Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.193753 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.193801 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.194218 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.194095 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949978bd-024c-4b88-849f-e89bd7f191d8-istiod-ca-cert" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:13.195242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.194301 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-socket" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:13.195242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.194335 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-certs" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:13.196138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.196109 5610 cpu_manager.go:414] "containerMap: removing container" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" Oct 29 22:40:13.196138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.196137 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" Oct 29 22:40:13.196530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.196407 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" containerName="istio-proxy" Oct 29 22:40:13.219124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.219079 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-envoy" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:13.219306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.219282 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-istio-token" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:13.219503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.219374 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/949978bd-024c-4b88-849f-e89bd7f191d8-istio-podinfo" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:13.219591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.219500 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-kube-api-access-pcpwn" pod "949978bd-024c-4b88-849f-e89bd7f191d8" (UID: "949978bd-024c-4b88-849f-e89bd7f191d8"). InnerVolumeSpecName "kube-api-access-pcpwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:13.233515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.232024 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj"] Oct 29 22:40:13.233515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.232171 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.237514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.237488 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr"] Oct 29 22:40:13.240661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.240621 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:13.240771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.240755 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:13.240874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.240848 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:13.296697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.296667 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:13.299171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299093 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.299311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299195 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.299311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299231 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hfz9\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-kube-api-access-8hfz9\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.299429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299339 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.299429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299377 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.299524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299450 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.299603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299585 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.299678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299663 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.300008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.299985 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786348c4-3d9b-4d2b-8094-f9f395568a8a-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.300302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.300285 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcpwn\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-kube-api-access-pcpwn\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.300434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.300420 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.317465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.317436 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949978bd-024c-4b88-849f-e89bd7f191d8-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.317715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.317697 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949978bd-024c-4b88-849f-e89bd7f191d8-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.320771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.320718 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.321023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.321006 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949978bd-024c-4b88-849f-e89bd7f191d8-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.324814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.324796 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949978bd-024c-4b88-849f-e89bd7f191d8-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.429649 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf"] Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.430900 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.430985 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.431075 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786348c4-3d9b-4d2b-8094-f9f395568a8a-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.431175 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.431250 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.431285 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hfz9\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-kube-api-access-8hfz9\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.431405 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.431450 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.431487 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.431988 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-credential-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.432241 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-data\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.432492 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-certs\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.433108 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786348c4-3d9b-4d2b-8094-f9f395568a8a-istiod-ca-cert\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.433345 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-socket\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.441179 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv"] Oct 29 22:40:13.470438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.447860 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-podinfo\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.509783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.509749 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.511292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.510703 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hfz9\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-kube-api-access-8hfz9\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.511292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.511122 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-token\") pod \"e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.559534 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.559511 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:13.650510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.650484 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f583580-6274-4723-8d24-931c393a76c4" path="/var/lib/kubelet/pods/1f583580-6274-4723-8d24-931c393a76c4/volumes" Oct 29 22:40:13.651717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.651483 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3017d7fa-514b-426a-b489-a34942e99745" path="/var/lib/kubelet/pods/3017d7fa-514b-426a-b489-a34942e99745/volumes" Oct 29 22:40:13.735223 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:13.735183 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:40:13.789728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.788878 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" event={"ID":"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6","Type":"ContainerStarted","Data":"78f3d83c12cdefd19e2e1c78cb4b374cc8fc2f20d254817de5f825b4b1281057"} Oct 29 22:40:13.789728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.788937 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" event={"ID":"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6","Type":"ContainerStarted","Data":"34a01be6184a982d1c47350329baa6d78f7063e913d91f1cdb8dd2ac4a7f3ac8"} Oct 29 22:40:13.792040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.791234 5610 generic.go:356] "Generic (PLEG): container finished" podID="949978bd-024c-4b88-849f-e89bd7f191d8" containerID="f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3" exitCode=0 Oct 29 22:40:13.792040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.791353 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" event={"ID":"949978bd-024c-4b88-849f-e89bd7f191d8","Type":"ContainerDied","Data":"f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3"} Oct 29 22:40:13.792040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.791411 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" event={"ID":"949978bd-024c-4b88-849f-e89bd7f191d8","Type":"ContainerDied","Data":"1488db3f7066eadc9fe3bc60353f6d36c9e9e808d7a1b9ac33d6e8086d244087"} Oct 29 22:40:13.792040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.791442 5610 scope.go:122] "RemoveContainer" containerID="f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3" Oct 29 22:40:13.792040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.791594 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47" Oct 29 22:40:13.796779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.796752 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" event={"ID":"171987f3-1f45-4938-a66f-702a5b025b7b","Type":"ContainerStarted","Data":"6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0"} Oct 29 22:40:13.796888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.796823 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" event={"ID":"171987f3-1f45-4938-a66f-702a5b025b7b","Type":"ContainerStarted","Data":"18b9b98fb5df34b4ea5e674b45dd6ed16809a21da1464ee7a721e81bca650437"} Oct 29 22:40:13.797134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.797091 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:13.804470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.804455 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9005-bbf3c9"] Oct 29 22:40:13.805865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.805816 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" event={"ID":"209974ec-e547-4d58-884b-1d13784bdabf","Type":"ContainerStarted","Data":"28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045"} Oct 29 22:40:13.806104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.806043 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" event={"ID":"209974ec-e547-4d58-884b-1d13784bdabf","Type":"ContainerStarted","Data":"d0093981344363782f2213c81b7ae1e6a1619454f313763a186096b0d033a7c6"} Oct 29 22:40:13.806569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.806537 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:13.810890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.810867 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9005-bbf3c9"] Oct 29 22:40:13.811948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.811932 5610 scope.go:122] "RemoveContainer" containerID="f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3" Oct 29 22:40:13.812328 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:13.812295 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3\": container with ID starting with f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3 not found: ID does not exist" containerID="f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3" Oct 29 22:40:13.812328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.812317 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3"} err="failed to get container status \"f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3\": rpc error: code = NotFound desc = could not find container \"f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3\": container with ID starting with f1680c0708998146554adc5413acb6bd479bc4c56e83b2cc23336a0145e4dfc3 not found: ID does not exist" Oct 29 22:40:13.839504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.839429 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" podStartSLOduration=67.83933241 podStartE2EDuration="1m7.83933241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:13.835899229 +0000 UTC m=+1476.317216737" watchObservedRunningTime="2026-10-29 22:40:13.83933241 +0000 UTC m=+1476.320649905" Oct 29 22:40:13.849025 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.848996 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:13.884065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.884005 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" podStartSLOduration=9.883987605 podStartE2EDuration="9.883987605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:13.875941153 +0000 UTC m=+1476.357258645" watchObservedRunningTime="2026-10-29 22:40:13.883987605 +0000 UTC m=+1476.365305100" Oct 29 22:40:13.893176 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:13.893112 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786348c4_3d9b_4d2b_8094_f9f395568a8a.slice/crio-cb87e6c8a3ad39c189456a6e271ea3464697535018f36453c0a82d7b2a48ea97 WatchSource:0}: Error finding container cb87e6c8a3ad39c189456a6e271ea3464697535018f36453c0a82d7b2a48ea97: Status 404 returned error can't find the container with id cb87e6c8a3ad39c189456a6e271ea3464697535018f36453c0a82d7b2a48ea97 Oct 29 22:40:13.896181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.896156 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj"] Oct 29 22:40:13.896784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.896756 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:13.896888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.896831 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:13.896888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.896878 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:13.916449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.916425 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47"] Oct 29 22:40:13.921471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.921449 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-s8k47"] Oct 29 22:40:13.958784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.958741 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" podStartSLOduration=8.958727898 podStartE2EDuration="8.958727898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:13.955473233 +0000 UTC m=+1476.436790728" watchObservedRunningTime="2026-10-29 22:40:13.958727898 +0000 UTC m=+1476.440045390" Oct 29 22:40:13.966821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.966790 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9005-2ae7b8"] Oct 29 22:40:13.968091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.968066 5610 cpu_manager.go:414] "containerMap: removing container" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" containerName="test-curl-9005-bbf3c9" Oct 29 22:40:13.968206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.968092 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" containerName="test-curl-9005-bbf3c9" Oct 29 22:40:13.968412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.968377 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" containerName="test-curl-9005-bbf3c9" Oct 29 22:40:13.974443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.974383 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:13.985933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:13.985908 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-2ae7b8"] Oct 29 22:40:14.056556 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.056517 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnvf8\" (UniqueName: \"kubernetes.io/projected/5a8ecee0-9468-4f45-9260-91ecafb80f7e-kube-api-access-lnvf8\") pod \"test-curl-9005-2ae7b8\" (UID: \"5a8ecee0-9468-4f45-9260-91ecafb80f7e\") " pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:14.159621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.159168 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvf8\" (UniqueName: \"kubernetes.io/projected/5a8ecee0-9468-4f45-9260-91ecafb80f7e-kube-api-access-lnvf8\") pod \"test-curl-9005-2ae7b8\" (UID: \"5a8ecee0-9468-4f45-9260-91ecafb80f7e\") " pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:14.209414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.207832 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvf8\" (UniqueName: \"kubernetes.io/projected/5a8ecee0-9468-4f45-9260-91ecafb80f7e-kube-api-access-lnvf8\") pod \"test-curl-9005-2ae7b8\" (UID: \"5a8ecee0-9468-4f45-9260-91ecafb80f7e\") " pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:14.318480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.307316 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:14.355641 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.355598 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" podUID="73580531-7db0-4a6a-9f0d-14ac98b530bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:14.735474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.735448 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:40:14.805450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.803838 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-2ae7b8"] Oct 29 22:40:14.805450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.804924 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-data\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.805450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.805004 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-credential-socket\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.805450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.805051 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-socket\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.805450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.805150 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-certs\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.805450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.805246 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83986232-6de2-4f0c-84f4-527002396e43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83986232-6de2-4f0c-84f4-527002396e43-istio-podinfo\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.805450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.805277 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-istio-token\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-istio-token\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.806900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.806106 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83986232-6de2-4f0c-84f4-527002396e43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83986232-6de2-4f0c-84f4-527002396e43-istiod-ca-cert\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.806900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.806184 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-kube-api-access-mm65h\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-kube-api-access-mm65h\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.806900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.806323 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-envoy\") pod \"83986232-6de2-4f0c-84f4-527002396e43\" (UID: \"83986232-6de2-4f0c-84f4-527002396e43\") " Oct 29 22:40:14.806900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.806412 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-socket" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:14.806900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.806619 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-data" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:14.807211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.807148 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-credential-socket" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:14.807266 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:14.807223 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a8ecee0_9468_4f45_9260_91ecafb80f7e.slice/crio-6760a8620cbf0b232b1b1b4d83a7d3b810f6eac046bdd47c5a2ea741496ef094 WatchSource:0}: Error finding container 6760a8620cbf0b232b1b1b4d83a7d3b810f6eac046bdd47c5a2ea741496ef094: Status 404 returned error can't find the container with id 6760a8620cbf0b232b1b1b4d83a7d3b810f6eac046bdd47c5a2ea741496ef094 Oct 29 22:40:14.807713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.807376 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-certs" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:14.809455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.808919 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83986232-6de2-4f0c-84f4-527002396e43-istiod-ca-cert" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.829711 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-kube-api-access-mm65h" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "kube-api-access-mm65h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.829797 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83986232-6de2-4f0c-84f4-527002396e43-istio-podinfo" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.842827 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg"] Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.846872 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.846912 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.846927 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83986232-6de2-4f0c-84f4-527002396e43-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.846944 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83986232-6de2-4f0c-84f4-527002396e43-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.846960 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm65h\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-kube-api-access-mm65h\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.846979 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.847002 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.847815 5610 cpu_manager.go:414] "containerMap: removing container" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.847838 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.848258 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="83986232-6de2-4f0c-84f4-527002396e43" containerName="istio-proxy" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.848768 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-istio-token" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.851154 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:40:14.909567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.851204 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 29 22:40:14.926321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.926273 5610 generic.go:356] "Generic (PLEG): container finished" podID="83986232-6de2-4f0c-84f4-527002396e43" containerID="6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29" exitCode=0 Oct 29 22:40:14.926779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.926739 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" Oct 29 22:40:14.935728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.935477 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9005-bbf3c9" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" containerName="test-curl-9005-bbf3c9" containerID="cri-o://77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453" gracePeriod=2 Oct 29 22:40:14.940105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.940077 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-envoy" pod "83986232-6de2-4f0c-84f4-527002396e43" (UID: "83986232-6de2-4f0c-84f4-527002396e43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:14.941061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.940792 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg"] Oct 29 22:40:14.941061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.940857 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" event={"ID":"83986232-6de2-4f0c-84f4-527002396e43","Type":"ContainerDied","Data":"6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29"} Oct 29 22:40:14.941061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.940890 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2" event={"ID":"83986232-6de2-4f0c-84f4-527002396e43","Type":"ContainerDied","Data":"50e2141cca7776477c9c779932b880902f5b0b4db01f00e20479aec557c629f9"} Oct 29 22:40:14.941061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.940935 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" event={"ID":"786348c4-3d9b-4d2b-8094-f9f395568a8a","Type":"ContainerStarted","Data":"c7bc8f6340b104e658a57608a5a3b3e36d5a638ecc161838e157468edf5506a2"} Oct 29 22:40:14.941061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.940960 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" event={"ID":"786348c4-3d9b-4d2b-8094-f9f395568a8a","Type":"ContainerStarted","Data":"cb87e6c8a3ad39c189456a6e271ea3464697535018f36453c0a82d7b2a48ea97"} Oct 29 22:40:14.941337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.941111 5610 scope.go:122] "RemoveContainer" containerID="6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29" Oct 29 22:40:14.941337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.941232 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:14.948602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.948545 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83986232-6de2-4f0c-84f4-527002396e43-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.948602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.948575 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83986232-6de2-4f0c-84f4-527002396e43-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:14.954109 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:14.954084 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-bbf3c9\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:14.957229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:14.956835 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a558b3-openshift-default-dockercfg-q7ghg\"" Oct 29 22:40:15.008215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.008192 5610 scope.go:122] "RemoveContainer" containerID="6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29" Oct 29 22:40:15.008875 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:15.008833 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29\": container with ID starting with 6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29 not found: ID does not exist" containerID="6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29" Oct 29 22:40:15.009081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.009052 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29"} err="failed to get container status \"6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29\": rpc error: code = NotFound desc = could not find container \"6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29\": container with ID starting with 6902c997640ce63889a04894339c22f6b95585f55eec0429c608f0f5cbe19b29 not found: ID does not exist" Oct 29 22:40:15.050532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.050506 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-token\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.050860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.050778 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.051149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.051113 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-data\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.051242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.051159 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nchj9\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-kube-api-access-nchj9\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.051300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.051244 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-certs\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.051376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.051355 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974c4bbd-66b9-46ff-9533-ee5c20c71313-istiod-ca-cert\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.051530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.051516 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-envoy\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.051652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.051592 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-podinfo\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.051951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.051898 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-credential-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.053228 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.053192 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" podStartSLOduration=69.053180363 podStartE2EDuration="1m9.053180363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:15.048267706 +0000 UTC m=+1477.529585201" watchObservedRunningTime="2026-10-29 22:40:15.053180363 +0000 UTC m=+1477.534497899" Oct 29 22:40:15.153528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153486 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.153705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153544 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-data\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.153705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153564 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nchj9\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-kube-api-access-nchj9\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.153705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153590 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-certs\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.153705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153632 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974c4bbd-66b9-46ff-9533-ee5c20c71313-istiod-ca-cert\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.153832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153717 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-envoy\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.153832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153772 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-podinfo\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.153893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153834 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-credential-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.153924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.153885 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-token\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.154221 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.154199 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-certs\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.154487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.154449 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.154742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.154706 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-credential-socket\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.154742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.154721 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-data\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.154973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.154952 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974c4bbd-66b9-46ff-9533-ee5c20c71313-istiod-ca-cert\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.156481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.156452 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-envoy\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.156964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.156946 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-podinfo\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.187596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.187547 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-token\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.210231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.210091 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:15.221830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.221799 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nchj9\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-kube-api-access-nchj9\") pod \"e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.326738 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.326698 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:15.341258 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:15.335632 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-bbf3c9\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:15.358446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.358348 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2"] Oct 29 22:40:15.442943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.427725 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-0d12c5-openshift-default-7cf588cbc7-dkcd2"] Oct 29 22:40:15.572485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.571811 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:15.572485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.571955 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:40:15.572485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.572002 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 29 22:40:15.646238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.646208 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83986232-6de2-4f0c-84f4-527002396e43" path="/var/lib/kubelet/pods/83986232-6de2-4f0c-84f4-527002396e43/volumes" Oct 29 22:40:15.647305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.647281 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949978bd-024c-4b88-849f-e89bd7f191d8" path="/var/lib/kubelet/pods/949978bd-024c-4b88-849f-e89bd7f191d8/volumes" Oct 29 22:40:15.849977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.849897 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:40:15.850447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.849966 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 29 22:40:15.932418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.931352 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg"] Oct 29 22:40:15.947285 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:15.947260 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974c4bbd_66b9_46ff_9533_ee5c20c71313.slice/crio-3e28e5b058334156f72dc93edd6cac8b64a20a3adac98cee4cbf4e15453a51de WatchSource:0}: Error finding container 3e28e5b058334156f72dc93edd6cac8b64a20a3adac98cee4cbf4e15453a51de: Status 404 returned error can't find the container with id 3e28e5b058334156f72dc93edd6cac8b64a20a3adac98cee4cbf4e15453a51de Oct 29 22:40:15.947992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.947970 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-2ae7b8" event={"ID":"5a8ecee0-9468-4f45-9260-91ecafb80f7e","Type":"ContainerStarted","Data":"ebf1f90b9d51a662ee22aef198bd1be81fb662dff177e8e46c36b6fb0e9afd67"} Oct 29 22:40:15.948122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.948109 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-2ae7b8" event={"ID":"5a8ecee0-9468-4f45-9260-91ecafb80f7e","Type":"ContainerStarted","Data":"6760a8620cbf0b232b1b1b4d83a7d3b810f6eac046bdd47c5a2ea741496ef094"} Oct 29 22:40:15.950936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.950575 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:15.951115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.951095 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:15.951203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:15.951192 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:40:16.031692 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:16.031637 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9005-2ae7b8" podStartSLOduration=3.03161345 podStartE2EDuration="3.03161345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:16.011602863 +0000 UTC m=+1478.492920359" watchObservedRunningTime="2026-10-29 22:40:16.03161345 +0000 UTC m=+1478.512930943" Oct 29 22:40:16.579073 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:16.579036 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:40:16.580207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:16.580172 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 29 22:40:16.849972 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:16.849900 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:40:16.850204 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:16.850179 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 29 22:40:16.988419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:16.969293 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" event={"ID":"974c4bbd-66b9-46ff-9533-ee5c20c71313","Type":"ContainerStarted","Data":"d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2"} Oct 29 22:40:16.988798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:16.988700 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" event={"ID":"974c4bbd-66b9-46ff-9533-ee5c20c71313","Type":"ContainerStarted","Data":"3e28e5b058334156f72dc93edd6cac8b64a20a3adac98cee4cbf4e15453a51de"} Oct 29 22:40:17.041360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.041314 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podStartSLOduration=71.041295041 podStartE2EDuration="1m11.041295041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:17.035073577 +0000 UTC m=+1479.516391073" watchObservedRunningTime="2026-10-29 22:40:17.041295041 +0000 UTC m=+1479.522612534" Oct 29 22:40:17.094160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.094073 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:17.145622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.145465 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9005-2ae7b8"] Oct 29 22:40:17.353788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.353759 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:17.489519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.455500 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9005-2ae7b8"] Oct 29 22:40:17.489519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.463229 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:17.540135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.533137 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9005-a8a6ca"] Oct 29 22:40:17.540135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.534519 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5a8ecee0-9468-4f45-9260-91ecafb80f7e" containerName="test-curl-9005-2ae7b8" Oct 29 22:40:17.540135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.534543 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8ecee0-9468-4f45-9260-91ecafb80f7e" containerName="test-curl-9005-2ae7b8" Oct 29 22:40:17.540135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.534792 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8ecee0-9468-4f45-9260-91ecafb80f7e" containerName="test-curl-9005-2ae7b8" Oct 29 22:40:17.540762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.540167 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:17.541576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.541549 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-a8a6ca"] Oct 29 22:40:17.655431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.654946 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:40:17.655431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.655009 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 29 22:40:17.729941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.729909 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhzdd\" (UniqueName: \"kubernetes.io/projected/32be8245-a97f-4da3-b556-4cfbb18be577-kube-api-access-fhzdd\") pod \"test-curl-9005-a8a6ca\" (UID: \"32be8245-a97f-4da3-b556-4cfbb18be577\") " pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:17.842900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.842421 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzdd\" (UniqueName: \"kubernetes.io/projected/32be8245-a97f-4da3-b556-4cfbb18be577-kube-api-access-fhzdd\") pod \"test-curl-9005-a8a6ca\" (UID: \"32be8245-a97f-4da3-b556-4cfbb18be577\") " pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:17.850118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.849803 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:40:17.850118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.849852 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 29 22:40:17.916699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.876170 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzdd\" (UniqueName: \"kubernetes.io/projected/32be8245-a97f-4da3-b556-4cfbb18be577-kube-api-access-fhzdd\") pod \"test-curl-9005-a8a6ca\" (UID: \"32be8245-a97f-4da3-b556-4cfbb18be577\") " pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:17.961126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:17.961095 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:17.977444 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:17.977260 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-bbf3c9\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:18.017793 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.017757 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:18.033095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.032757 5610 generic.go:356] "Generic (PLEG): container finished" podID="446d1b8e-11af-4aef-b589-a48a14d842b6" containerID="77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453" exitCode=137 Oct 29 22:40:18.033095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.032840 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:18.033095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.032943 5610 scope.go:122] "RemoveContainer" containerID="77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453" Oct 29 22:40:18.033095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.033066 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9005-2ae7b8" podUID="5a8ecee0-9468-4f45-9260-91ecafb80f7e" containerName="test-curl-9005-2ae7b8" containerID="cri-o://ebf1f90b9d51a662ee22aef198bd1be81fb662dff177e8e46c36b6fb0e9afd67" gracePeriod=2 Oct 29 22:40:18.033450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.033433 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:18.034957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.034938 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:40:18.046588 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:18.046549 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-bbf3c9\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:18.048497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.048477 5610 scope.go:122] "RemoveContainer" containerID="77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453" Oct 29 22:40:18.048760 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.048736 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446d1b8e-11af-4aef-b589-a48a14d842b6-kube-api-access-xlwnv\" (UniqueName: \"kubernetes.io/projected/446d1b8e-11af-4aef-b589-a48a14d842b6-kube-api-access-xlwnv\") pod \"446d1b8e-11af-4aef-b589-a48a14d842b6\" (UID: \"446d1b8e-11af-4aef-b589-a48a14d842b6\") " Oct 29 22:40:18.048948 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:18.048819 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453\": container with ID starting with 77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453 not found: ID does not exist" containerID="77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453" Oct 29 22:40:18.049144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.049124 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453"} err="failed to get container status \"77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453\": rpc error: code = NotFound desc = could not find container \"77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453\": container with ID starting with 77a58bc46b11b4fef6fd5718cb98988be81b608dea5d4b0f3da81d5c62cff453 not found: ID does not exist" Oct 29 22:40:18.051632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.051375 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446d1b8e-11af-4aef-b589-a48a14d842b6-kube-api-access-xlwnv" pod "446d1b8e-11af-4aef-b589-a48a14d842b6" (UID: "446d1b8e-11af-4aef-b589-a48a14d842b6"). InnerVolumeSpecName "kube-api-access-xlwnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:18.051822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.051794 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlwnv\" (UniqueName: \"kubernetes.io/projected/446d1b8e-11af-4aef-b589-a48a14d842b6-kube-api-access-xlwnv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:18.080422 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:18.059534 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-bbf3c9\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:18.391418 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:18.377518 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-bbf3c9\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:18.536868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.532802 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-a8a6ca"] Oct 29 22:40:18.548602 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:18.548573 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32be8245_a97f_4da3_b556_4cfbb18be577.slice/crio-bd0a333b66aa12805957cc7423df56947f4e775bb364d2ec9d29cb16786a8710 WatchSource:0}: Error finding container bd0a333b66aa12805957cc7423df56947f4e775bb364d2ec9d29cb16786a8710: Status 404 returned error can't find the container with id bd0a333b66aa12805957cc7423df56947f4e775bb364d2ec9d29cb16786a8710 Oct 29 22:40:18.597328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.597291 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:40:18.597641 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.597608 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 29 22:40:18.849379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.849281 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:40:18.849379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:18.849358 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 29 22:40:19.059193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.059161 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-a8a6ca" event={"ID":"32be8245-a97f-4da3-b556-4cfbb18be577","Type":"ContainerStarted","Data":"c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1"} Oct 29 22:40:19.078970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.078935 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-a8a6ca" event={"ID":"32be8245-a97f-4da3-b556-4cfbb18be577","Type":"ContainerStarted","Data":"bd0a333b66aa12805957cc7423df56947f4e775bb364d2ec9d29cb16786a8710"} Oct 29 22:40:19.130476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.129956 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9005-a8a6ca" podStartSLOduration=2.129933876 podStartE2EDuration="2.129933876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:19.099697188 +0000 UTC m=+1481.581014682" watchObservedRunningTime="2026-10-29 22:40:19.129933876 +0000 UTC m=+1481.611251370" Oct 29 22:40:19.131737 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:19.131702 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-bbf3c9\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" pod="opendatahub/test-curl-9005-bbf3c9" Oct 29 22:40:19.306414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.304861 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:19.568237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.568190 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:40:19.568559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.568515 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 29 22:40:19.610435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.609627 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:19.693842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.693804 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446d1b8e-11af-4aef-b589-a48a14d842b6" path="/var/lib/kubelet/pods/446d1b8e-11af-4aef-b589-a48a14d842b6/volumes" Oct 29 22:40:19.796455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.795223 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:19.849996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.849907 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 29 22:40:19.849996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:19.849974 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 29 22:40:20.128181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.128093 5610 generic.go:356] "Generic (PLEG): container finished" podID="5a8ecee0-9468-4f45-9260-91ecafb80f7e" containerID="ebf1f90b9d51a662ee22aef198bd1be81fb662dff177e8e46c36b6fb0e9afd67" exitCode=137 Oct 29 22:40:20.428481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.427627 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9005-a8a6ca"] Oct 29 22:40:20.433429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.432302 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9005-a8a6ca"] Oct 29 22:40:20.514858 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.514666 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:20.534119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.534092 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9005-2ede4e"] Oct 29 22:40:20.535109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.535090 5610 cpu_manager.go:414] "containerMap: removing container" podUID="32be8245-a97f-4da3-b556-4cfbb18be577" containerName="test-curl-9005-a8a6ca" Oct 29 22:40:20.535209 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.535199 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32be8245-a97f-4da3-b556-4cfbb18be577" containerName="test-curl-9005-a8a6ca" Oct 29 22:40:20.535480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.535468 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="32be8245-a97f-4da3-b556-4cfbb18be577" containerName="test-curl-9005-a8a6ca" Oct 29 22:40:20.537050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.537032 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8ecee0-9468-4f45-9260-91ecafb80f7e-kube-api-access-lnvf8\" (UniqueName: \"kubernetes.io/projected/5a8ecee0-9468-4f45-9260-91ecafb80f7e-kube-api-access-lnvf8\") pod \"5a8ecee0-9468-4f45-9260-91ecafb80f7e\" (UID: \"5a8ecee0-9468-4f45-9260-91ecafb80f7e\") " Oct 29 22:40:20.539276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.539237 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8ecee0-9468-4f45-9260-91ecafb80f7e-kube-api-access-lnvf8" pod "5a8ecee0-9468-4f45-9260-91ecafb80f7e" (UID: "5a8ecee0-9468-4f45-9260-91ecafb80f7e"). InnerVolumeSpecName "kube-api-access-lnvf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:20.539570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.539554 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:20.557565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.557369 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-2ede4e"] Oct 29 22:40:20.580607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.571213 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 29 22:40:20.580607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.571264 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 29 22:40:20.639481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.639292 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwdcl\" (UniqueName: \"kubernetes.io/projected/8f05c4e4-b8a6-41be-ae7b-200de30de3e7-kube-api-access-mwdcl\") pod \"test-curl-9005-2ede4e\" (UID: \"8f05c4e4-b8a6-41be-ae7b-200de30de3e7\") " pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:20.640912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.640890 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnvf8\" (UniqueName: \"kubernetes.io/projected/5a8ecee0-9468-4f45-9260-91ecafb80f7e-kube-api-access-lnvf8\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:20.742307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.742270 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwdcl\" (UniqueName: \"kubernetes.io/projected/8f05c4e4-b8a6-41be-ae7b-200de30de3e7-kube-api-access-mwdcl\") pod \"test-curl-9005-2ede4e\" (UID: \"8f05c4e4-b8a6-41be-ae7b-200de30de3e7\") " pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:20.777043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.777008 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwdcl\" (UniqueName: \"kubernetes.io/projected/8f05c4e4-b8a6-41be-ae7b-200de30de3e7-kube-api-access-mwdcl\") pod \"test-curl-9005-2ede4e\" (UID: \"8f05c4e4-b8a6-41be-ae7b-200de30de3e7\") " pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:20.854533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.854499 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:20.854857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.854832 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:20.855826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.855807 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:40:20.913716 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:20.913677 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:21.143803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.143771 5610 scope.go:122] "RemoveContainer" containerID="ebf1f90b9d51a662ee22aef198bd1be81fb662dff177e8e46c36b6fb0e9afd67" Oct 29 22:40:21.144213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.143867 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9005-a8a6ca" podUID="32be8245-a97f-4da3-b556-4cfbb18be577" containerName="test-curl-9005-a8a6ca" containerID="cri-o://c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1" gracePeriod=2 Oct 29 22:40:21.144213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.144117 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:21.157695 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:21.157667 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-a8a6ca\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="32be8245-a97f-4da3-b556-4cfbb18be577" pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:21.170642 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:21.170595 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-a8a6ca\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="32be8245-a97f-4da3-b556-4cfbb18be577" pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:21.183697 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:21.183657 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-2ae7b8\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="5a8ecee0-9468-4f45-9260-91ecafb80f7e" pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:21.281515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.281442 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-2ede4e"] Oct 29 22:40:21.283005 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:21.282940 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f05c4e4_b8a6_41be_ae7b_200de30de3e7.slice/crio-3eb32fcdaba681f7e00b0e050f917b33c48233e65de96e4307447c5eb9d3e9ba WatchSource:0}: Error finding container 3eb32fcdaba681f7e00b0e050f917b33c48233e65de96e4307447c5eb9d3e9ba: Status 404 returned error can't find the container with id 3eb32fcdaba681f7e00b0e050f917b33c48233e65de96e4307447c5eb9d3e9ba Oct 29 22:40:21.322445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.321234 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:21.434437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.434387 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk"] Oct 29 22:40:21.439360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.439342 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:21.452542 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:21.452512 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-a8a6ca\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="32be8245-a97f-4da3-b556-4cfbb18be577" pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:21.454377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.454349 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdxx\" (UniqueName: \"kubernetes.io/projected/73b700f4-19cf-4aaa-885a-f328c224d5c4-kube-api-access-qwdxx\") pod \"payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk\" (UID: \"73b700f4-19cf-4aaa-885a-f328c224d5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:21.454731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.454712 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b700f4-19cf-4aaa-885a-f328c224d5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk\" (UID: \"73b700f4-19cf-4aaa-885a-f328c224d5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:21.457205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.457185 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a558b3-dockercfg-4q2w4\"" Oct 29 22:40:21.459040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.458559 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a558b3\"" Oct 29 22:40:21.459147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.459086 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk"] Oct 29 22:40:21.465348 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:21.465322 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-2ae7b8\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="5a8ecee0-9468-4f45-9260-91ecafb80f7e" pod="opendatahub/test-curl-9005-2ae7b8" Oct 29 22:40:21.467101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.467079 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:21.628482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.612481 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b700f4-19cf-4aaa-885a-f328c224d5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk\" (UID: \"73b700f4-19cf-4aaa-885a-f328c224d5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:21.628482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.612657 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdxx\" (UniqueName: \"kubernetes.io/projected/73b700f4-19cf-4aaa-885a-f328c224d5c4-kube-api-access-qwdxx\") pod \"payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk\" (UID: \"73b700f4-19cf-4aaa-885a-f328c224d5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:21.739417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.719315 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b700f4-19cf-4aaa-885a-f328c224d5c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk\" (UID: \"73b700f4-19cf-4aaa-885a-f328c224d5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:21.739417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.733107 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdxx\" (UniqueName: \"kubernetes.io/projected/73b700f4-19cf-4aaa-885a-f328c224d5c4-kube-api-access-qwdxx\") pod \"payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk\" (UID: \"73b700f4-19cf-4aaa-885a-f328c224d5c4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:21.764200 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.764167 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:21.796094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.796046 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8ecee0-9468-4f45-9260-91ecafb80f7e" path="/var/lib/kubelet/pods/5a8ecee0-9468-4f45-9260-91ecafb80f7e/volumes" Oct 29 22:40:21.796705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.796675 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:21.799028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.799002 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:21.804680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:21.804661 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:40:22.160542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:22.160465 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-2ede4e" event={"ID":"8f05c4e4-b8a6-41be-ae7b-200de30de3e7","Type":"ContainerStarted","Data":"4d392b5c7597d7dfa2d5bc02fc79905e1b5358c7905e0d4d22a9b41ba4bc9c44"} Oct 29 22:40:22.161138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:22.161097 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-2ede4e" event={"ID":"8f05c4e4-b8a6-41be-ae7b-200de30de3e7","Type":"ContainerStarted","Data":"3eb32fcdaba681f7e00b0e050f917b33c48233e65de96e4307447c5eb9d3e9ba"} Oct 29 22:40:22.213519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:22.213464 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9005-2ede4e" podStartSLOduration=2.213450299 podStartE2EDuration="2.213450299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:22.20913026 +0000 UTC m=+1484.690447752" watchObservedRunningTime="2026-10-29 22:40:22.213450299 +0000 UTC m=+1484.694767789" Oct 29 22:40:22.399405 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:22.399363 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b700f4_19cf_4aaa_885a_f328c224d5c4.slice/crio-60a6a5e8a06223b89864096715515663b85ae575c8aba31c8abaf426e26a11f1 WatchSource:0}: Error finding container 60a6a5e8a06223b89864096715515663b85ae575c8aba31c8abaf426e26a11f1: Status 404 returned error can't find the container with id 60a6a5e8a06223b89864096715515663b85ae575c8aba31c8abaf426e26a11f1 Oct 29 22:40:22.400238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:22.400209 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk"] Oct 29 22:40:23.087947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.087898 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9005-2ede4e"] Oct 29 22:40:23.093771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.093739 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9005-2ede4e"] Oct 29 22:40:23.202301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.202258 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" event={"ID":"73b700f4-19cf-4aaa-885a-f328c224d5c4","Type":"ContainerStarted","Data":"20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2"} Oct 29 22:40:23.202800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.202348 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" event={"ID":"73b700f4-19cf-4aaa-885a-f328c224d5c4","Type":"ContainerStarted","Data":"60a6a5e8a06223b89864096715515663b85ae575c8aba31c8abaf426e26a11f1"} Oct 29 22:40:23.202800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.202635 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:23.244542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.244476 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" podStartSLOduration=2.244458142 podStartE2EDuration="2.244458142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:23.240660933 +0000 UTC m=+1485.721978427" watchObservedRunningTime="2026-10-29 22:40:23.244458142 +0000 UTC m=+1485.725775636" Oct 29 22:40:23.423126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.423102 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:23.430436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.428262 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:23.478414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.462026 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32be8245-a97f-4da3-b556-4cfbb18be577-kube-api-access-fhzdd\" (UniqueName: \"kubernetes.io/projected/32be8245-a97f-4da3-b556-4cfbb18be577-kube-api-access-fhzdd\") pod \"32be8245-a97f-4da3-b556-4cfbb18be577\" (UID: \"32be8245-a97f-4da3-b556-4cfbb18be577\") " Oct 29 22:40:23.563548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.541582 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32be8245-a97f-4da3-b556-4cfbb18be577-kube-api-access-fhzdd" pod "32be8245-a97f-4da3-b556-4cfbb18be577" (UID: "32be8245-a97f-4da3-b556-4cfbb18be577"). InnerVolumeSpecName "kube-api-access-fhzdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:23.595463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.594579 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhzdd\" (UniqueName: \"kubernetes.io/projected/32be8245-a97f-4da3-b556-4cfbb18be577-kube-api-access-fhzdd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:23.639666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.639592 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32be8245-a97f-4da3-b556-4cfbb18be577" path="/var/lib/kubelet/pods/32be8245-a97f-4da3-b556-4cfbb18be577/volumes" Oct 29 22:40:23.811193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.811163 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:40:23.819134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.819105 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:23.987416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:23.975666 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:24.212557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.212523 5610 generic.go:356] "Generic (PLEG): container finished" podID="32be8245-a97f-4da3-b556-4cfbb18be577" containerID="c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1" exitCode=137 Oct 29 22:40:24.213027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.212567 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-a8a6ca" Oct 29 22:40:24.213027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.212608 5610 scope.go:122] "RemoveContainer" containerID="c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1" Oct 29 22:40:24.213027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.212956 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9005-2ede4e" podUID="8f05c4e4-b8a6-41be-ae7b-200de30de3e7" containerName="test-curl-9005-2ede4e" containerID="cri-o://4d392b5c7597d7dfa2d5bc02fc79905e1b5358c7905e0d4d22a9b41ba4bc9c44" gracePeriod=2 Oct 29 22:40:24.223041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.223021 5610 scope.go:122] "RemoveContainer" containerID="c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1" Oct 29 22:40:24.223309 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:24.223290 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1\": container with ID starting with c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1 not found: ID does not exist" containerID="c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1" Oct 29 22:40:24.223370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.223318 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1"} err="failed to get container status \"c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1\": rpc error: code = NotFound desc = could not find container \"c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1\": container with ID starting with c3fa4271b211fd6e819db754922b651725fe6d58b2161cf5664a5ff1bf041df1 not found: ID does not exist" Oct 29 22:40:24.347276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.347184 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" podUID="73580531-7db0-4a6a-9f0d-14ac98b530bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:40:24.939302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.939268 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:40:24.939528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:24.939376 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:40:25.015959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.015922 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:25.192737 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.188772 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:25.797727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.797701 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:40:25.857604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.857564 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf"] Oct 29 22:40:25.858657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.858634 5610 cpu_manager.go:414] "containerMap: removing container" podUID="73580531-7db0-4a6a-9f0d-14ac98b530bd" containerName="payload-processing" Oct 29 22:40:25.858657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.858656 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73580531-7db0-4a6a-9f0d-14ac98b530bd" containerName="payload-processing" Oct 29 22:40:25.858926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.858689 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8f05c4e4-b8a6-41be-ae7b-200de30de3e7" containerName="test-curl-9005-2ede4e" Oct 29 22:40:25.858926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.858697 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f05c4e4-b8a6-41be-ae7b-200de30de3e7" containerName="test-curl-9005-2ede4e" Oct 29 22:40:25.858926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.858854 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="73580531-7db0-4a6a-9f0d-14ac98b530bd" containerName="payload-processing" Oct 29 22:40:25.858926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.858869 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f05c4e4-b8a6-41be-ae7b-200de30de3e7" containerName="test-curl-9005-2ede4e" Oct 29 22:40:25.872663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.872636 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:25.882291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.881847 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73580531-7db0-4a6a-9f0d-14ac98b530bd-kube-api-access-ssxwz\" (UniqueName: \"kubernetes.io/projected/73580531-7db0-4a6a-9f0d-14ac98b530bd-kube-api-access-ssxwz\") pod \"73580531-7db0-4a6a-9f0d-14ac98b530bd\" (UID: \"73580531-7db0-4a6a-9f0d-14ac98b530bd\") " Oct 29 22:40:25.882291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.882058 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73580531-7db0-4a6a-9f0d-14ac98b530bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73580531-7db0-4a6a-9f0d-14ac98b530bd-plugins-config-volume\") pod \"73580531-7db0-4a6a-9f0d-14ac98b530bd\" (UID: \"73580531-7db0-4a6a-9f0d-14ac98b530bd\") " Oct 29 22:40:25.884104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.883675 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73580531-7db0-4a6a-9f0d-14ac98b530bd-plugins-config-volume" pod "73580531-7db0-4a6a-9f0d-14ac98b530bd" (UID: "73580531-7db0-4a6a-9f0d-14ac98b530bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:25.886959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.886932 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73580531-7db0-4a6a-9f0d-14ac98b530bd-kube-api-access-ssxwz" pod "73580531-7db0-4a6a-9f0d-14ac98b530bd" (UID: "73580531-7db0-4a6a-9f0d-14ac98b530bd"). InnerVolumeSpecName "kube-api-access-ssxwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:25.897470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.895173 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf"] Oct 29 22:40:25.984515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.984487 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1336dcd8-9214-4694-b955-c3fd3cc865ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf\" (UID: \"1336dcd8-9214-4694-b955-c3fd3cc865ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:25.984671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.984547 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62lc\" (UniqueName: \"kubernetes.io/projected/1336dcd8-9214-4694-b955-c3fd3cc865ec-kube-api-access-v62lc\") pod \"payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf\" (UID: \"1336dcd8-9214-4694-b955-c3fd3cc865ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:25.984863 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.984841 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxwz\" (UniqueName: \"kubernetes.io/projected/73580531-7db0-4a6a-9f0d-14ac98b530bd-kube-api-access-ssxwz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:25.984904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:25.984867 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73580531-7db0-4a6a-9f0d-14ac98b530bd-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:26.086240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.086144 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1336dcd8-9214-4694-b955-c3fd3cc865ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf\" (UID: \"1336dcd8-9214-4694-b955-c3fd3cc865ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:26.086240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.086231 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62lc\" (UniqueName: \"kubernetes.io/projected/1336dcd8-9214-4694-b955-c3fd3cc865ec-kube-api-access-v62lc\") pod \"payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf\" (UID: \"1336dcd8-9214-4694-b955-c3fd3cc865ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:26.087162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.087134 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1336dcd8-9214-4694-b955-c3fd3cc865ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf\" (UID: \"1336dcd8-9214-4694-b955-c3fd3cc865ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:26.105406 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.105371 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62lc\" (UniqueName: \"kubernetes.io/projected/1336dcd8-9214-4694-b955-c3fd3cc865ec-kube-api-access-v62lc\") pod \"payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf\" (UID: \"1336dcd8-9214-4694-b955-c3fd3cc865ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:26.218008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.217978 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:26.338945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.338747 5610 generic.go:356] "Generic (PLEG): container finished" podID="8f05c4e4-b8a6-41be-ae7b-200de30de3e7" containerID="4d392b5c7597d7dfa2d5bc02fc79905e1b5358c7905e0d4d22a9b41ba4bc9c44" exitCode=137 Oct 29 22:40:26.346497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.346101 5610 generic.go:356] "Generic (PLEG): container finished" podID="73580531-7db0-4a6a-9f0d-14ac98b530bd" containerID="6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51" exitCode=0 Oct 29 22:40:26.346497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.346262 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" event={"ID":"73580531-7db0-4a6a-9f0d-14ac98b530bd","Type":"ContainerDied","Data":"6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51"} Oct 29 22:40:26.346497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.346315 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" event={"ID":"73580531-7db0-4a6a-9f0d-14ac98b530bd","Type":"ContainerDied","Data":"1ad739adca001d31326e1fd84dea74f15a923a503d20b9efc5c24c355151cdc1"} Oct 29 22:40:26.346497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.346347 5610 scope.go:122] "RemoveContainer" containerID="6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51" Oct 29 22:40:26.346497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.346354 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n" Oct 29 22:40:26.364891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.364868 5610 scope.go:122] "RemoveContainer" containerID="6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51" Oct 29 22:40:26.365281 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:26.365250 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51\": container with ID starting with 6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51 not found: ID does not exist" containerID="6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51" Oct 29 22:40:26.365381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.365293 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51"} err="failed to get container status \"6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51\": rpc error: code = NotFound desc = could not find container \"6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51\": container with ID starting with 6ca70b478f3eca1e6c34815332ec2d986cb208020e15f42b6aba5af7a5a39a51 not found: ID does not exist" Oct 29 22:40:26.408895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.408851 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n"] Oct 29 22:40:26.416607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.416199 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-p468n"] Oct 29 22:40:26.526021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.525984 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf"] Oct 29 22:40:26.527478 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:26.527448 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1336dcd8_9214_4694_b955_c3fd3cc865ec.slice/crio-8ffa33cbadbc8cbc0e2f61f4c293d678908f4fd53c56f87d6c37dc92a0068b79 WatchSource:0}: Error finding container 8ffa33cbadbc8cbc0e2f61f4c293d678908f4fd53c56f87d6c37dc92a0068b79: Status 404 returned error can't find the container with id 8ffa33cbadbc8cbc0e2f61f4c293d678908f4fd53c56f87d6c37dc92a0068b79 Oct 29 22:40:26.548905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.548875 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:26.696640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.696605 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f05c4e4-b8a6-41be-ae7b-200de30de3e7-kube-api-access-mwdcl\" (UniqueName: \"kubernetes.io/projected/8f05c4e4-b8a6-41be-ae7b-200de30de3e7-kube-api-access-mwdcl\") pod \"8f05c4e4-b8a6-41be-ae7b-200de30de3e7\" (UID: \"8f05c4e4-b8a6-41be-ae7b-200de30de3e7\") " Oct 29 22:40:26.698787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.698756 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f05c4e4-b8a6-41be-ae7b-200de30de3e7-kube-api-access-mwdcl" pod "8f05c4e4-b8a6-41be-ae7b-200de30de3e7" (UID: "8f05c4e4-b8a6-41be-ae7b-200de30de3e7"). InnerVolumeSpecName "kube-api-access-mwdcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:26.799621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:26.799583 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwdcl\" (UniqueName: \"kubernetes.io/projected/8f05c4e4-b8a6-41be-ae7b-200de30de3e7-kube-api-access-mwdcl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:27.031416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.028669 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:27.356135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.356055 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:27.356135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.356072 5610 scope.go:122] "RemoveContainer" containerID="4d392b5c7597d7dfa2d5bc02fc79905e1b5358c7905e0d4d22a9b41ba4bc9c44" Oct 29 22:40:27.364254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.364223 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" event={"ID":"1336dcd8-9214-4694-b955-c3fd3cc865ec","Type":"ContainerStarted","Data":"08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547"} Oct 29 22:40:27.364418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.364284 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" event={"ID":"1336dcd8-9214-4694-b955-c3fd3cc865ec","Type":"ContainerStarted","Data":"8ffa33cbadbc8cbc0e2f61f4c293d678908f4fd53c56f87d6c37dc92a0068b79"} Oct 29 22:40:27.364418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.364335 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:27.404118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.404064 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" podStartSLOduration=6.404051448 podStartE2EDuration="6.404051448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:27.401163692 +0000 UTC m=+1489.882481188" watchObservedRunningTime="2026-10-29 22:40:27.404051448 +0000 UTC m=+1489.885368940" Oct 29 22:40:27.414132 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:27.414105 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-2ede4e\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="8f05c4e4-b8a6-41be-ae7b-200de30de3e7" pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:27.662471 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:27.657523 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9005-2ede4e\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="8f05c4e4-b8a6-41be-ae7b-200de30de3e7" pod="opendatahub/test-curl-9005-2ede4e" Oct 29 22:40:27.662471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.657656 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73580531-7db0-4a6a-9f0d-14ac98b530bd" path="/var/lib/kubelet/pods/73580531-7db0-4a6a-9f0d-14ac98b530bd/volumes" Oct 29 22:40:27.662471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:27.660416 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f05c4e4-b8a6-41be-ae7b-200de30de3e7" path="/var/lib/kubelet/pods/8f05c4e4-b8a6-41be-ae7b-200de30de3e7/volumes" Oct 29 22:40:28.627865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:28.627828 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:28.767430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:28.766241 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:30.533788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:30.533739 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:32.222470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:32.222433 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:32.371551 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:32.371439 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:32.518652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:32.518563 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:34.057408 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:34.057307 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:34.216362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:34.216332 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:40:35.538443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:35.538411 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9"] Oct 29 22:40:35.538849 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:35.538608 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" containerID="cri-o://eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef" gracePeriod=30 Oct 29 22:40:35.550135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:35.550102 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt"] Oct 29 22:40:35.550368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:35.550339 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" containerID="cri-o://f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1" gracePeriod=30 Oct 29 22:40:35.554862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:35.554840 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c"] Oct 29 22:40:35.555119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:35.555091 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" containerID="cri-o://bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644" gracePeriod=30 Oct 29 22:40:35.780417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:35.777652 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:37.100907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:37.100875 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:37.253910 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:37.253866 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:37.809947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:37.809913 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:38.383662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:38.383635 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:40:38.764208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:38.764016 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:40.437634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.437595 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:40.838890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.838867 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:40:40.913842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.913805 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq"] Oct 29 22:40:40.914772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.914750 5610 cpu_manager.go:414] "containerMap: removing container" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" Oct 29 22:40:40.914772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.914770 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" Oct 29 22:40:40.915045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.915030 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerName="istio-proxy" Oct 29 22:40:40.918794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.918776 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:40.924450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.924221 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-socket\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.924450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.924276 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-data\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.924637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.924510 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-socket" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:40.924637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.924561 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-data" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:40.924753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.924642 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-certs" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:40.924843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.924384 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-certs\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.925021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.925004 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-envoy\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.925083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.925050 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-kube-api-access-8xqpb\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-kube-api-access-8xqpb\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.925146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.925093 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-token\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-token\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.925206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.925146 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-credential-socket\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.925206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.925179 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-podinfo\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.925317 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.925306 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924894a6-42ae-41e5-b760-71a6ea4ac86f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924894a6-42ae-41e5-b760-71a6ea4ac86f-istiod-ca-cert\") pod \"924894a6-42ae-41e5-b760-71a6ea4ac86f\" (UID: \"924894a6-42ae-41e5-b760-71a6ea4ac86f\") " Oct 29 22:40:40.926106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.926067 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924894a6-42ae-41e5-b760-71a6ea4ac86f-istiod-ca-cert" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:40.926726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.926651 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924894a6-42ae-41e5-b760-71a6ea4ac86f-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:40.926726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.926677 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:40.926726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.926690 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:40.927237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.927213 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-credential-socket" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:40.927659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.926705 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:40.928427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.928383 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-envoy" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:40.932185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.932154 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-kube-api-access-8xqpb" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "kube-api-access-8xqpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:40.932185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.932183 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-token" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:40.932445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.932424 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-podinfo" pod "924894a6-42ae-41e5-b760-71a6ea4ac86f" (UID: "924894a6-42ae-41e5-b760-71a6ea4ac86f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:40.935134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.935097 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-433712-serving-cert\"" Oct 29 22:40:40.941447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:40.939696 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq"] Oct 29 22:40:41.029808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.029718 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnw9\" (UniqueName: \"kubernetes.io/projected/cb1b41de-a916-4233-8219-737eaf1377a7-kube-api-access-7jnw9\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.029986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.029920 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b41de-a916-4233-8219-737eaf1377a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.029986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.029968 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-maas-api-tls\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.030180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.030164 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-metrics-certs\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.030375 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.030361 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.030455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.030380 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xqpb\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-kube-api-access-8xqpb\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.030455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.030432 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.030455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.030444 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924894a6-42ae-41e5-b760-71a6ea4ac86f-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.030455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.030452 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924894a6-42ae-41e5-b760-71a6ea4ac86f-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.132276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.132238 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-metrics-certs\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.132501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.132462 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnw9\" (UniqueName: \"kubernetes.io/projected/cb1b41de-a916-4233-8219-737eaf1377a7-kube-api-access-7jnw9\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.132629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.132603 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b41de-a916-4233-8219-737eaf1377a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.132714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.132645 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-maas-api-tls\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.133515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.133470 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b41de-a916-4233-8219-737eaf1377a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.135485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.135190 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-maas-api-tls\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.135485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.135473 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-metrics-certs\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.153900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.153872 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnw9\" (UniqueName: \"kubernetes.io/projected/cb1b41de-a916-4233-8219-737eaf1377a7-kube-api-access-7jnw9\") pod \"maas-api-e2e-api-b-433712-579589d4dc-d7nkq\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.276937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.275727 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:41.287626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.283611 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:40:41.327146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.327118 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:40:41.336937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336346 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-socket\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.336937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336501 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-certs\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.336937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336672 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-istio-token\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-istio-token\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.336937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336714 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-credential-socket\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.336937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336743 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b52788e-737b-4c5b-ac84-22329e92373a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52788e-737b-4c5b-ac84-22329e92373a-istiod-ca-cert\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.336937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336844 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-kube-api-access-clq5n\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-kube-api-access-clq5n\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.336937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336884 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-envoy\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.336937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336916 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-data\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.337609 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.336957 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b52788e-737b-4c5b-ac84-22329e92373a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52788e-737b-4c5b-ac84-22329e92373a-istio-podinfo\") pod \"6b52788e-737b-4c5b-ac84-22329e92373a\" (UID: \"6b52788e-737b-4c5b-ac84-22329e92373a\") " Oct 29 22:40:41.339384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.338509 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-credential-socket" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.339384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.339281 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b52788e-737b-4c5b-ac84-22329e92373a-istiod-ca-cert" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:41.339384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.339342 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-certs" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.339384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.339072 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-data" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.339717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.339432 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-socket" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.341631 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.341575 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.341631 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.341601 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.341951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.341909 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.342118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.342065 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.342292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.342086 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b52788e-737b-4c5b-ac84-22329e92373a-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.354353 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.354291 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd"] Oct 29 22:40:41.355912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.355886 5610 cpu_manager.go:414] "containerMap: removing container" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" Oct 29 22:40:41.357094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.357074 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" Oct 29 22:40:41.357248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.357236 5610 cpu_manager.go:414] "containerMap: removing container" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" Oct 29 22:40:41.357335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.357322 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" Oct 29 22:40:41.357710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.357697 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerName="istio-proxy" Oct 29 22:40:41.357800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.357792 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" containerName="istio-proxy" Oct 29 22:40:41.362003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.361986 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.380498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.380459 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c06e05-serving-cert\"" Oct 29 22:40:41.394796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.394745 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd"] Oct 29 22:40:41.400268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.400237 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-istio-token" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:41.402262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.400520 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-envoy" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.402262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.400643 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-kube-api-access-clq5n" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "kube-api-access-clq5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:41.402262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.402226 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b52788e-737b-4c5b-ac84-22329e92373a-istio-podinfo" pod "6b52788e-737b-4c5b-ac84-22329e92373a" (UID: "6b52788e-737b-4c5b-ac84-22329e92373a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:41.443916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.443881 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-socket\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.443963 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-data\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444003 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d78288-6f8f-4815-a748-ba548bd1d97e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d78288-6f8f-4815-a748-ba548bd1d97e-istiod-ca-cert\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444042 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-credential-socket\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444087 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-certs\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444152 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-kube-api-access-4cjt9\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-kube-api-access-4cjt9\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444194 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-podinfo\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444196 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-socket" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444232 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-token\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-token\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444310 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-envoy\") pod \"95d78288-6f8f-4815-a748-ba548bd1d97e\" (UID: \"95d78288-6f8f-4815-a748-ba548bd1d97e\") " Oct 29 22:40:41.444951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444488 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-credential-socket" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.444951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444629 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce6012c-043e-4bcd-b986-180c8d2f38c0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.444951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444662 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-data" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.444951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444733 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-metrics-certs\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.444951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444842 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwm44\" (UniqueName: \"kubernetes.io/projected/1ce6012c-043e-4bcd-b986-180c8d2f38c0-kube-api-access-hwm44\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.444951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444908 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-maas-api-tls\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.444951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.444922 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d78288-6f8f-4815-a748-ba548bd1d97e-istiod-ca-cert" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445100 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-certs" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445266 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445289 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445303 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clq5n\" (UniqueName: \"kubernetes.io/projected/6b52788e-737b-4c5b-ac84-22329e92373a-kube-api-access-clq5n\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445317 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b52788e-737b-4c5b-ac84-22329e92373a-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445330 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b52788e-737b-4c5b-ac84-22329e92373a-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445344 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445357 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445372 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d78288-6f8f-4815-a748-ba548bd1d97e-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.445838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.445382 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.448230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.448169 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-podinfo" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:40:41.448350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.448235 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-kube-api-access-4cjt9" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "kube-api-access-4cjt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:41.448898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.448862 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-envoy" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:40:41.449492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.449467 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-token" pod "95d78288-6f8f-4815-a748-ba548bd1d97e" (UID: "95d78288-6f8f-4815-a748-ba548bd1d97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:40:41.547385 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.547285 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwm44\" (UniqueName: \"kubernetes.io/projected/1ce6012c-043e-4bcd-b986-180c8d2f38c0-kube-api-access-hwm44\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.547555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.547442 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-maas-api-tls\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.547629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.547601 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce6012c-043e-4bcd-b986-180c8d2f38c0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.547733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.547710 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-metrics-certs\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.547878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.547862 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjt9\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-kube-api-access-4cjt9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.547942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.547884 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.547942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.547901 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.547942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.547916 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d78288-6f8f-4815-a748-ba548bd1d97e-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:40:41.548432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.548386 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce6012c-043e-4bcd-b986-180c8d2f38c0-openshift-service-ca\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.550438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.550409 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-metrics-certs\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.550513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.550452 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-maas-api-tls\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.563528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.563503 5610 generic.go:356] "Generic (PLEG): container finished" podID="95d78288-6f8f-4815-a748-ba548bd1d97e" containerID="f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1" exitCode=0 Oct 29 22:40:41.563676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.563558 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" Oct 29 22:40:41.563676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.563596 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" event={"ID":"95d78288-6f8f-4815-a748-ba548bd1d97e","Type":"ContainerDied","Data":"f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1"} Oct 29 22:40:41.563847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.563706 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt" event={"ID":"95d78288-6f8f-4815-a748-ba548bd1d97e","Type":"ContainerDied","Data":"ed1d898e6cd5a69ff55d1b1389e841b64f2351d217f85d1f30f06386cb23b85b"} Oct 29 22:40:41.563847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.563741 5610 scope.go:122] "RemoveContainer" containerID="f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1" Oct 29 22:40:41.566141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.565633 5610 generic.go:356] "Generic (PLEG): container finished" podID="924894a6-42ae-41e5-b760-71a6ea4ac86f" containerID="eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef" exitCode=0 Oct 29 22:40:41.566141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.565765 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" event={"ID":"924894a6-42ae-41e5-b760-71a6ea4ac86f","Type":"ContainerDied","Data":"eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef"} Oct 29 22:40:41.566141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.565803 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" event={"ID":"924894a6-42ae-41e5-b760-71a6ea4ac86f","Type":"ContainerDied","Data":"4fc088919d46a15cd3f18165bfbf6325966aac7fc857b7de4e7baeaa58e2fe1a"} Oct 29 22:40:41.566141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.565873 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9" Oct 29 22:40:41.570423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.568793 5610 generic.go:356] "Generic (PLEG): container finished" podID="6b52788e-737b-4c5b-ac84-22329e92373a" containerID="bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644" exitCode=0 Oct 29 22:40:41.570423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.568827 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" event={"ID":"6b52788e-737b-4c5b-ac84-22329e92373a","Type":"ContainerDied","Data":"bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644"} Oct 29 22:40:41.570423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.568887 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" event={"ID":"6b52788e-737b-4c5b-ac84-22329e92373a","Type":"ContainerDied","Data":"b9306da14f2c37e0a257dcdb9db12878dcd77f16434c28de5bb66f7217b815f3"} Oct 29 22:40:41.570423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.568834 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c" Oct 29 22:40:41.575362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.575335 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwm44\" (UniqueName: \"kubernetes.io/projected/1ce6012c-043e-4bcd-b986-180c8d2f38c0-kube-api-access-hwm44\") pod \"maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.577445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.577372 5610 scope.go:122] "RemoveContainer" containerID="f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1" Oct 29 22:40:41.577909 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:41.577777 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1\": container with ID starting with f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1 not found: ID does not exist" containerID="f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1" Oct 29 22:40:41.577909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.577810 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1"} err="failed to get container status \"f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1\": rpc error: code = NotFound desc = could not find container \"f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1\": container with ID starting with f58e18a9962fef1ac9506b173651bd414e170e405a35bd44d6cc662b83798bf1 not found: ID does not exist" Oct 29 22:40:41.577909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.577834 5610 scope.go:122] "RemoveContainer" containerID="eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef" Oct 29 22:40:41.621433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.620579 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq"] Oct 29 22:40:41.621552 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:41.621536 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1b41de_a916_4233_8219_737eaf1377a7.slice/crio-ad023bcd0c9cfddc66fd162e9d69a8714cb09602d5a42966c71895f8f77fed72 WatchSource:0}: Error finding container ad023bcd0c9cfddc66fd162e9d69a8714cb09602d5a42966c71895f8f77fed72: Status 404 returned error can't find the container with id ad023bcd0c9cfddc66fd162e9d69a8714cb09602d5a42966c71895f8f77fed72 Oct 29 22:40:41.623206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.623189 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:40:41.643699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.643671 5610 scope.go:122] "RemoveContainer" containerID="eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef" Oct 29 22:40:41.643995 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:41.643974 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef\": container with ID starting with eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef not found: ID does not exist" containerID="eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef" Oct 29 22:40:41.644054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.644004 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef"} err="failed to get container status \"eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef\": rpc error: code = NotFound desc = could not find container \"eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef\": container with ID starting with eea6e155285c076a8efe2dd07a0c55b2029565f17dc0c2ae77a028fd6002bbef not found: ID does not exist" Oct 29 22:40:41.644054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.644028 5610 scope.go:122] "RemoveContainer" containerID="bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644" Oct 29 22:40:41.655177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.655156 5610 scope.go:122] "RemoveContainer" containerID="bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644" Oct 29 22:40:41.655511 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:41.655492 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644\": container with ID starting with bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644 not found: ID does not exist" containerID="bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644" Oct 29 22:40:41.655568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.655522 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644"} err="failed to get container status \"bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644\": rpc error: code = NotFound desc = could not find container \"bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644\": container with ID starting with bd0d9089d8595f13be0c25bb6584469b1a3802e075a4653673f1f4cd41c33644 not found: ID does not exist" Oct 29 22:40:41.684232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.684199 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9"] Oct 29 22:40:41.690563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.690533 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-fldg9"] Oct 29 22:40:41.715586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.715550 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:41.718532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.718503 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt"] Oct 29 22:40:41.722801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.722778 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-zhxjt"] Oct 29 22:40:41.757899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.757868 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c"] Oct 29 22:40:41.764260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:41.764233 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-7l56c"] Oct 29 22:40:42.027802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.027775 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd"] Oct 29 22:40:42.028948 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:42.028922 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce6012c_043e_4bcd_b986_180c8d2f38c0.slice/crio-4718f60532ac03397212e71bf4d27c6d0bb49430b684302b8e785d7c3a171c17 WatchSource:0}: Error finding container 4718f60532ac03397212e71bf4d27c6d0bb49430b684302b8e785d7c3a171c17: Status 404 returned error can't find the container with id 4718f60532ac03397212e71bf4d27c6d0bb49430b684302b8e785d7c3a171c17 Oct 29 22:40:42.130765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.130736 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:42.582282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.582247 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" event={"ID":"1ce6012c-043e-4bcd-b986-180c8d2f38c0","Type":"ContainerStarted","Data":"1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440"} Oct 29 22:40:42.582762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.582296 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" event={"ID":"1ce6012c-043e-4bcd-b986-180c8d2f38c0","Type":"ContainerStarted","Data":"4718f60532ac03397212e71bf4d27c6d0bb49430b684302b8e785d7c3a171c17"} Oct 29 22:40:42.582762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.582608 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:42.585344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.585314 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" event={"ID":"cb1b41de-a916-4233-8219-737eaf1377a7","Type":"ContainerStarted","Data":"a382c916ac220d16242d4b67e92ec98dfa4265edae67315cd602be002fddf4ee"} Oct 29 22:40:42.585498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.585358 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" event={"ID":"cb1b41de-a916-4233-8219-737eaf1377a7","Type":"ContainerStarted","Data":"ad023bcd0c9cfddc66fd162e9d69a8714cb09602d5a42966c71895f8f77fed72"} Oct 29 22:40:42.585498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.585447 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:42.625159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:42.625096 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" podStartSLOduration=10.200164442 podStartE2EDuration="10.625081769s" totalImagesPullingTime="424.917327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:42.622641401 +0000 UTC m=+1505.103958895" watchObservedRunningTime="2026-10-29 22:40:42.625081769 +0000 UTC m=+1505.106399261" Oct 29 22:40:43.629335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:43.629294 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:43.637933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:43.637907 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b52788e-737b-4c5b-ac84-22329e92373a" path="/var/lib/kubelet/pods/6b52788e-737b-4c5b-ac84-22329e92373a/volumes" Oct 29 22:40:43.638429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:43.638411 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924894a6-42ae-41e5-b760-71a6ea4ac86f" path="/var/lib/kubelet/pods/924894a6-42ae-41e5-b760-71a6ea4ac86f/volumes" Oct 29 22:40:43.638898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:43.638883 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d78288-6f8f-4815-a748-ba548bd1d97e" path="/var/lib/kubelet/pods/95d78288-6f8f-4815-a748-ba548bd1d97e/volumes" Oct 29 22:40:43.711307 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:40:43.711273 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:40:43.788992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:43.788951 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:44.737661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:44.737623 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:45.239546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.239497 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" podStartSLOduration=16.948761073 podStartE2EDuration="17.239481543s" totalImagesPullingTime="290.72047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:42.663465882 +0000 UTC m=+1505.144783375" watchObservedRunningTime="2026-10-29 22:40:45.239481543 +0000 UTC m=+1507.720799033" Oct 29 22:40:45.242383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.242356 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz"] Oct 29 22:40:45.248792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.248771 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.272099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.272071 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz"] Oct 29 22:40:45.276605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.276579 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1ae775-serving-cert\"" Oct 29 22:40:45.323239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.323207 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc7877-05b4-4d34-b99d-f81c24397ec0-openshift-service-ca\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.323447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.323243 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvt9\" (UniqueName: \"kubernetes.io/projected/8bcc7877-05b4-4d34-b99d-f81c24397ec0-kube-api-access-hnvt9\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.323525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.323490 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-maas-api-tls\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.323634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.323522 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-metrics-certs\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.373011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.372972 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:45.424948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.424866 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc7877-05b4-4d34-b99d-f81c24397ec0-openshift-service-ca\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.424948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.424902 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvt9\" (UniqueName: \"kubernetes.io/projected/8bcc7877-05b4-4d34-b99d-f81c24397ec0-kube-api-access-hnvt9\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.425126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.425051 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-maas-api-tls\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.425126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.425068 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-metrics-certs\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.425684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.425658 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc7877-05b4-4d34-b99d-f81c24397ec0-openshift-service-ca\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.427604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.427582 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-metrics-certs\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.427698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.427613 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-maas-api-tls\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.444706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.444655 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvt9\" (UniqueName: \"kubernetes.io/projected/8bcc7877-05b4-4d34-b99d-f81c24397ec0-kube-api-access-hnvt9\") pod \"maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.568729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.568689 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:45.574957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.574912 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:45.955410 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:45.955364 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz"] Oct 29 22:40:45.955810 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:40:45.955782 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcc7877_05b4_4d34_b99d_f81c24397ec0.slice/crio-d97f1bed0c2a37ab6852ca525d4b27a5d402b07e6bc2a3a170f530402ea0515b WatchSource:0}: Error finding container d97f1bed0c2a37ab6852ca525d4b27a5d402b07e6bc2a3a170f530402ea0515b: Status 404 returned error can't find the container with id d97f1bed0c2a37ab6852ca525d4b27a5d402b07e6bc2a3a170f530402ea0515b Oct 29 22:40:46.710746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:46.709357 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" event={"ID":"8bcc7877-05b4-4d34-b99d-f81c24397ec0","Type":"ContainerStarted","Data":"e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3"} Oct 29 22:40:46.710746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:46.709477 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" event={"ID":"8bcc7877-05b4-4d34-b99d-f81c24397ec0","Type":"ContainerStarted","Data":"d97f1bed0c2a37ab6852ca525d4b27a5d402b07e6bc2a3a170f530402ea0515b"} Oct 29 22:40:46.710746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:46.710651 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:46.751697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:46.751647 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" podStartSLOduration=1.444245065 podStartE2EDuration="1.751629864s" totalImagesPullingTime="307.384799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:40:46.748575451 +0000 UTC m=+1509.229892945" watchObservedRunningTime="2026-10-29 22:40:46.751629864 +0000 UTC m=+1509.232947356" Oct 29 22:40:47.157094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:47.157009 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:47.316419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:47.313910 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:48.604152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:48.604118 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:40:48.605216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:48.605191 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:40:48.960780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:48.960744 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:50.628323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:50.628284 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:52.249592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:52.249545 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:53.739443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:53.739384 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:40:53.947594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:53.947545 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:55.199994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:55.199953 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:55.350418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:55.350093 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:56.071866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:56.071826 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:40:57.926533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:57.926483 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:40:58.866071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:40:58.866031 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:00.595255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:00.595219 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:00.726677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:00.726639 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:00.866300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:00.866199 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:02.328057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:02.328023 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:03.955554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:03.955515 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:05.415807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:05.415772 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:06.873516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:06.873477 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:08.514905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:08.514869 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:10.149417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:10.147636 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:11.913266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:11.913211 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:13.660107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:13.660074 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:13.803692 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:41:13.803638 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:41:14.880925 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:14.880887 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9"] Oct 29 22:41:14.881492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:14.881152 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" podUID="8644924f-1343-4fde-a1f5-e919ab41e380" containerName="maas-api" containerID="cri-o://281d975ad03562c6b0653f499e8a60a9a872d688f61853f626b03ad9179dc490" gracePeriod=30 Oct 29 22:41:14.984749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:14.984714 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:15.839213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.839175 5610 generic.go:356] "Generic (PLEG): container finished" podID="8644924f-1343-4fde-a1f5-e919ab41e380" containerID="281d975ad03562c6b0653f499e8a60a9a872d688f61853f626b03ad9179dc490" exitCode=0 Oct 29 22:41:15.840205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.840174 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" event={"ID":"8644924f-1343-4fde-a1f5-e919ab41e380","Type":"ContainerDied","Data":"281d975ad03562c6b0653f499e8a60a9a872d688f61853f626b03ad9179dc490"} Oct 29 22:41:15.840483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.840387 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" event={"ID":"8644924f-1343-4fde-a1f5-e919ab41e380","Type":"ContainerDied","Data":"3982343a7ea016357d50c3e3b7ccccfeab8334732353be1ffab75e4917295561"} Oct 29 22:41:15.840634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.840618 5610 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3982343a7ea016357d50c3e3b7ccccfeab8334732353be1ffab75e4917295561" Oct 29 22:41:15.852877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.852841 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:41:15.922894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.922856 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm"] Oct 29 22:41:15.924744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.924714 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8644924f-1343-4fde-a1f5-e919ab41e380" containerName="maas-api" Oct 29 22:41:15.924744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.924740 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8644924f-1343-4fde-a1f5-e919ab41e380" containerName="maas-api" Oct 29 22:41:15.924920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.924911 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8644924f-1343-4fde-a1f5-e919ab41e380" containerName="maas-api" Oct 29 22:41:15.928792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.928772 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:15.943123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.943101 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a558b3-serving-cert\"" Oct 29 22:41:15.978037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.968359 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8644924f-1343-4fde-a1f5-e919ab41e380-kube-api-access-xhbwd\" (UniqueName: \"kubernetes.io/projected/8644924f-1343-4fde-a1f5-e919ab41e380-kube-api-access-xhbwd\") pod \"8644924f-1343-4fde-a1f5-e919ab41e380\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " Oct 29 22:41:15.991559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.991523 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8644924f-1343-4fde-a1f5-e919ab41e380-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8644924f-1343-4fde-a1f5-e919ab41e380-openshift-service-ca\") pod \"8644924f-1343-4fde-a1f5-e919ab41e380\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " Oct 29 22:41:15.991810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.991789 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls\") pod \"8644924f-1343-4fde-a1f5-e919ab41e380\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " Oct 29 22:41:15.992350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.992329 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-metrics-certs\") pod \"8644924f-1343-4fde-a1f5-e919ab41e380\" (UID: \"8644924f-1343-4fde-a1f5-e919ab41e380\") " Oct 29 22:41:15.993628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.993598 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8644924f-1343-4fde-a1f5-e919ab41e380-openshift-service-ca" pod "8644924f-1343-4fde-a1f5-e919ab41e380" (UID: "8644924f-1343-4fde-a1f5-e919ab41e380"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:15.995561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:15.995538 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm"] Oct 29 22:41:16.002549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.002501 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-metrics-certs" pod "8644924f-1343-4fde-a1f5-e919ab41e380" (UID: "8644924f-1343-4fde-a1f5-e919ab41e380"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:16.002682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.002668 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls" pod "8644924f-1343-4fde-a1f5-e919ab41e380" (UID: "8644924f-1343-4fde-a1f5-e919ab41e380"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:41:16.003239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.003101 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8644924f-1343-4fde-a1f5-e919ab41e380-kube-api-access-xhbwd" pod "8644924f-1343-4fde-a1f5-e919ab41e380" (UID: "8644924f-1343-4fde-a1f5-e919ab41e380"). InnerVolumeSpecName "kube-api-access-xhbwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:16.095859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.095745 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52d3f42-c8b7-4377-a22c-921b0b66e237-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.096038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.095897 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-maas-api-tls\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.096038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.096004 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw62c\" (UniqueName: \"kubernetes.io/projected/f52d3f42-c8b7-4377-a22c-921b0b66e237-kube-api-access-pw62c\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.096195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.096177 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-metrics-certs\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.096367 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.096351 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbwd\" (UniqueName: \"kubernetes.io/projected/8644924f-1343-4fde-a1f5-e919ab41e380-kube-api-access-xhbwd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:41:16.096468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.096373 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8644924f-1343-4fde-a1f5-e919ab41e380-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:41:16.096468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.096404 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:41:16.096468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.096419 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8644924f-1343-4fde-a1f5-e919ab41e380-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:41:16.198295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.198247 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-maas-api-tls\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.198546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.198370 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw62c\" (UniqueName: \"kubernetes.io/projected/f52d3f42-c8b7-4377-a22c-921b0b66e237-kube-api-access-pw62c\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.198546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.198509 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-metrics-certs\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.198749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.198619 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52d3f42-c8b7-4377-a22c-921b0b66e237-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.199274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.199250 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52d3f42-c8b7-4377-a22c-921b0b66e237-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.209438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.203982 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-metrics-certs\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.210358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.210325 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-maas-api-tls\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.213757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.213711 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:16.217428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.217409 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw62c\" (UniqueName: \"kubernetes.io/projected/f52d3f42-c8b7-4377-a22c-921b0b66e237-kube-api-access-pw62c\") pod \"maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.247642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.247616 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:16.373494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.371799 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:16.514417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.513939 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:16.583166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.583133 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm"] Oct 29 22:41:16.595121 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:41:16.595090 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52d3f42_c8b7_4377_a22c_921b0b66e237.slice/crio-5b5eb0a6737fa6c88eb4d99d6ca064c7731d68df69973ab90c249a76459c6d98 WatchSource:0}: Error finding container 5b5eb0a6737fa6c88eb4d99d6ca064c7731d68df69973ab90c249a76459c6d98: Status 404 returned error can't find the container with id 5b5eb0a6737fa6c88eb4d99d6ca064c7731d68df69973ab90c249a76459c6d98 Oct 29 22:41:16.724489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.724455 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:16.852973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.852938 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" event={"ID":"f52d3f42-c8b7-4377-a22c-921b0b66e237","Type":"ContainerStarted","Data":"5b5eb0a6737fa6c88eb4d99d6ca064c7731d68df69973ab90c249a76459c6d98"} Oct 29 22:41:16.853142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.853036 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9" Oct 29 22:41:16.910166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.910134 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9"] Oct 29 22:41:16.916084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:16.916053 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-685v9"] Oct 29 22:41:17.639047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:17.639010 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8644924f-1343-4fde-a1f5-e919ab41e380" path="/var/lib/kubelet/pods/8644924f-1343-4fde-a1f5-e919ab41e380/volumes" Oct 29 22:41:17.759343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:17.759306 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:17.862902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:17.862860 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" event={"ID":"f52d3f42-c8b7-4377-a22c-921b0b66e237","Type":"ContainerStarted","Data":"e6b1cc0f13dec373fd388a6bfee48620df7ab14e542d0a8b69d0cacf29120d1f"} Oct 29 22:41:17.863122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:17.862958 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:17.911745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:17.911633 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" podStartSLOduration=17.551422625 podStartE2EDuration="17.911614688s" totalImagesPullingTime="360.192063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:17.906986605 +0000 UTC m=+1540.388304098" watchObservedRunningTime="2026-10-29 22:41:17.911614688 +0000 UTC m=+1540.392932183" Oct 29 22:41:18.357415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:18.353766 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:20.055514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:20.055387 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:20.553035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:20.552999 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:21.786362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:21.785890 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:23.564887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:23.564759 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:23.880606 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:23.880525 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:41:23.966799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:23.965723 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:24.103545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:24.103504 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:24.850099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:24.850066 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:25.018510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:25.018454 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:25.800988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:25.800939 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:27.422834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:27.422793 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:28.556425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:28.556380 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw"] Oct 29 22:41:28.733666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:28.733629 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:28.876124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:28.875980 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:29.255874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:29.255841 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:30.041320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:30.041278 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:30.193095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:30.193047 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:31.815530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:31.815489 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:32.302121 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:32.302071 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:32.947677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:32.947636 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:33.413361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:33.413240 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:33.576414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:33.567487 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" podUID="5fef595a-be02-49f5-9287-1f0f47359337" containerName="payload-processing" containerID="cri-o://59cd074a91c6c2f703e21ce9474446ba28171fbce461aae0d114680a349fd4cc" gracePeriod=25 Oct 29 22:41:33.595447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:33.595351 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:33.960127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:33.956750 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:34.150019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.149813 5610 generic.go:356] "Generic (PLEG): container finished" podID="5fef595a-be02-49f5-9287-1f0f47359337" containerID="59cd074a91c6c2f703e21ce9474446ba28171fbce461aae0d114680a349fd4cc" exitCode=0 Oct 29 22:41:34.151726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.150343 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" event={"ID":"5fef595a-be02-49f5-9287-1f0f47359337","Type":"ContainerDied","Data":"59cd074a91c6c2f703e21ce9474446ba28171fbce461aae0d114680a349fd4cc"} Oct 29 22:41:34.335381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.335347 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:41:34.377361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.377329 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fef595a-be02-49f5-9287-1f0f47359337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fef595a-be02-49f5-9287-1f0f47359337-plugins-config-volume\") pod \"5fef595a-be02-49f5-9287-1f0f47359337\" (UID: \"5fef595a-be02-49f5-9287-1f0f47359337\") " Oct 29 22:41:34.377668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.377477 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fef595a-be02-49f5-9287-1f0f47359337-kube-api-access-x84x9\" (UniqueName: \"kubernetes.io/projected/5fef595a-be02-49f5-9287-1f0f47359337-kube-api-access-x84x9\") pod \"5fef595a-be02-49f5-9287-1f0f47359337\" (UID: \"5fef595a-be02-49f5-9287-1f0f47359337\") " Oct 29 22:41:34.377846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.377819 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fef595a-be02-49f5-9287-1f0f47359337-plugins-config-volume" pod "5fef595a-be02-49f5-9287-1f0f47359337" (UID: "5fef595a-be02-49f5-9287-1f0f47359337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:41:34.378427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.378351 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fef595a-be02-49f5-9287-1f0f47359337-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:41:34.383574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.383480 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fef595a-be02-49f5-9287-1f0f47359337-kube-api-access-x84x9" pod "5fef595a-be02-49f5-9287-1f0f47359337" (UID: "5fef595a-be02-49f5-9287-1f0f47359337"). InnerVolumeSpecName "kube-api-access-x84x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:41:34.392762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.392732 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4"] Oct 29 22:41:34.394112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.394090 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5fef595a-be02-49f5-9287-1f0f47359337" containerName="payload-processing" Oct 29 22:41:34.394221 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.394114 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fef595a-be02-49f5-9287-1f0f47359337" containerName="payload-processing" Oct 29 22:41:34.394325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.394312 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fef595a-be02-49f5-9287-1f0f47359337" containerName="payload-processing" Oct 29 22:41:34.399139 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.399116 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:34.413192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.413148 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4"] Oct 29 22:41:34.417428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.417386 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d478b36e\"" Oct 29 22:41:34.417883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.417842 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d478b36e-dockercfg-g5dhv\"" Oct 29 22:41:34.480079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.480040 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xltp\" (UniqueName: \"kubernetes.io/projected/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-kube-api-access-2xltp\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4\" (UID: \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:34.480277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.480177 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4\" (UID: \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:34.480373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.480323 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x84x9\" (UniqueName: \"kubernetes.io/projected/5fef595a-be02-49f5-9287-1f0f47359337-kube-api-access-x84x9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:41:34.581803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.581756 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xltp\" (UniqueName: \"kubernetes.io/projected/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-kube-api-access-2xltp\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4\" (UID: \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:34.581992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.581866 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4\" (UID: \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:34.582637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.582612 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-plugins-config-volume\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4\" (UID: \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:34.602114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.602078 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xltp\" (UniqueName: \"kubernetes.io/projected/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-kube-api-access-2xltp\") pod \"payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4\" (UID: \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\") " pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:34.724918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:34.724886 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:35.040520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.040489 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4"] Oct 29 22:41:35.041885 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:41:35.041843 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d34b5d6_eb5c_4dc9_b6f8_8434d71331e6.slice/crio-2f018b8ce7664247f4fd3b00998cb63a3d679e1ddf064128c428c74b9d627450 WatchSource:0}: Error finding container 2f018b8ce7664247f4fd3b00998cb63a3d679e1ddf064128c428c74b9d627450: Status 404 returned error can't find the container with id 2f018b8ce7664247f4fd3b00998cb63a3d679e1ddf064128c428c74b9d627450 Oct 29 22:41:35.160189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.160152 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" event={"ID":"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6","Type":"ContainerStarted","Data":"5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369"} Oct 29 22:41:35.160363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.160212 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" event={"ID":"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6","Type":"ContainerStarted","Data":"2f018b8ce7664247f4fd3b00998cb63a3d679e1ddf064128c428c74b9d627450"} Oct 29 22:41:35.160363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.160248 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:35.162458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.162424 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" event={"ID":"5fef595a-be02-49f5-9287-1f0f47359337","Type":"ContainerDied","Data":"303c622ab94e3bcb76d8c2b523fd59f4927ef0465f592b62c487bd46e89c23dd"} Oct 29 22:41:35.162591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.162444 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw" Oct 29 22:41:35.162591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.162566 5610 scope.go:122] "RemoveContainer" containerID="59cd074a91c6c2f703e21ce9474446ba28171fbce461aae0d114680a349fd4cc" Oct 29 22:41:35.203654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.203593 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" podStartSLOduration=10.203575744 podStartE2EDuration="10.203575744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:35.196871356 +0000 UTC m=+1557.678188851" watchObservedRunningTime="2026-10-29 22:41:35.203575744 +0000 UTC m=+1557.684893241" Oct 29 22:41:35.232889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.232845 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw"] Oct 29 22:41:35.245540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.245510 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0ab33e-54f7dbfcc8-bsvsw"] Oct 29 22:41:35.640279 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.640241 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fef595a-be02-49f5-9287-1f0f47359337" path="/var/lib/kubelet/pods/5fef595a-be02-49f5-9287-1f0f47359337/volumes" Oct 29 22:41:35.990489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:35.990440 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:36.134828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:36.134794 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:37.695825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:37.695783 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:37.866384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:37.866338 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:38.013359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:38.013261 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:39.385361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.385314 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:39.410429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.408307 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb"] Oct 29 22:41:39.418831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.418799 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.437043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.437014 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb"] Oct 29 22:41:39.442222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.442194 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a26426\"/\"default-dockercfg-2lm9c\"" Oct 29 22:41:39.442385 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.442319 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a26426\"/\"test-model-a26426-kserve-self-signed-certs\"" Oct 29 22:41:39.449300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.449276 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a26426\"/\"kube-root-ca.crt\"" Oct 29 22:41:39.449463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.449324 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a26426\"/\"openshift-service-ca.crt\"" Oct 29 22:41:39.521083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.521047 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a268be3-9077-44bf-893f-13fe297eea66-tls-certs\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.521262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.521100 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-dshm\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.521262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.521189 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-home\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.521262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.521241 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjn6q\" (UniqueName: \"kubernetes.io/projected/4a268be3-9077-44bf-893f-13fe297eea66-kube-api-access-tjn6q\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.521599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.521572 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-model-cache\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.521667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.521611 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-tmp-dir\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.623371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.623331 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-model-cache\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.623371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.623377 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-tmp-dir\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.623681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.623505 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a268be3-9077-44bf-893f-13fe297eea66-tls-certs\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.623681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.623551 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-dshm\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.623681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.623594 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-home\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.623681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.623647 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn6q\" (UniqueName: \"kubernetes.io/projected/4a268be3-9077-44bf-893f-13fe297eea66-kube-api-access-tjn6q\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.624025 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.623752 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-model-cache\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.624145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.624113 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-home\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.624361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.624339 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-tmp-dir\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.626858 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.626820 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-dshm\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.627276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.627252 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a268be3-9077-44bf-893f-13fe297eea66-tls-certs\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.649535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.649438 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn6q\" (UniqueName: \"kubernetes.io/projected/4a268be3-9077-44bf-893f-13fe297eea66-kube-api-access-tjn6q\") pod \"test-model-a26426-kserve-9847889bf-l7nmb\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.758426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.758217 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:39.791117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:39.790927 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:40.037493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:40.016665 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:41.029008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:41.024604 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb"] Oct 29 22:41:41.642417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:41.626222 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" event={"ID":"4a268be3-9077-44bf-893f-13fe297eea66","Type":"ContainerStarted","Data":"2fcb39c505b9c34cfddcd389cf5f426e52ea5de0821e6437234b917a114efa95"} Oct 29 22:41:41.853590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:41.853480 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:42.267666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:42.267628 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:43.313987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:43.313951 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:43.650590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:43.650512 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" event={"ID":"4a268be3-9077-44bf-893f-13fe297eea66","Type":"ContainerStarted","Data":"2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f"} Oct 29 22:41:43.730175 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:43.729295 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podStartSLOduration=4.729278505 podStartE2EDuration="4.729278505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:43.715932839 +0000 UTC m=+1566.197250333" watchObservedRunningTime="2026-10-29 22:41:43.729278505 +0000 UTC m=+1566.210595998" Oct 29 22:41:43.758064 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:41:43.758026 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:41:43.759145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:43.759129 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:45.428802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:45.428687 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:46.217419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:46.216671 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:41:47.391890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:47.391852 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:49.074567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:49.074527 5610 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:41:49.074977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:49.074611 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:41:49.153107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:49.153067 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:49.759207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:49.759161 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:49.777007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:49.776973 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:49.808991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:49.808959 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:41:51.128277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:51.128244 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:51.651083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:51.651022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:52.159374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:52.159340 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:52.651808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:52.651775 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:53.151759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.151711 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:53.318414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.313365 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:53.900937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.900902 5610 generic.go:356] "Generic (PLEG): container finished" podID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerID="d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2" exitCode=0 Oct 29 22:41:53.901158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.900978 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" event={"ID":"974c4bbd-66b9-46ff-9533-ee5c20c71313","Type":"ContainerDied","Data":"d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2"} Oct 29 22:41:53.901574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.901545 5610 scope.go:122] "RemoveContainer" containerID="d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2" Oct 29 22:41:53.903385 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.903363 5610 generic.go:356] "Generic (PLEG): container finished" podID="a9943a5e-db25-4438-bdf0-74979290b12e" containerID="772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec" exitCode=0 Oct 29 22:41:53.903515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.903458 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" event={"ID":"a9943a5e-db25-4438-bdf0-74979290b12e","Type":"ContainerDied","Data":"772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec"} Oct 29 22:41:53.903923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.903905 5610 scope.go:122] "RemoveContainer" containerID="772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec" Oct 29 22:41:53.904073 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.904046 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:41:53.904146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.904130 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:41:53.904207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.904180 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:41:53.906075 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.906043 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:41:53.906159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.906123 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:41:53.906200 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:53.906174 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:41:54.939254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:54.936222 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:54.959169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:54.959132 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" event={"ID":"a9943a5e-db25-4438-bdf0-74979290b12e","Type":"ContainerStarted","Data":"ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85"} Oct 29 22:41:54.963916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:54.963886 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" event={"ID":"974c4bbd-66b9-46ff-9533-ee5c20c71313","Type":"ContainerStarted","Data":"a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb"} Oct 29 22:41:54.977578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:54.977552 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:41:55.330417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:55.329965 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:41:55.330417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:55.330153 5610 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 29 22:41:55.330417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:55.330199 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 29 22:41:55.997418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:55.996679 5610 patch_prober.go:29] interesting pod/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:41:55.997418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:55.996740 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 29 22:41:56.332510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:56.330293 5610 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 29 22:41:56.332510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:56.330369 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 29 22:41:56.889417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:56.887792 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:56.981418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:56.978810 5610 patch_prober.go:29] interesting pod/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:41:56.981418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:56.978930 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 29 22:41:57.329846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:57.329760 5610 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 29 22:41:57.329846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:57.329820 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 29 22:41:57.977809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:57.977770 5610 patch_prober.go:29] interesting pod/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:41:57.978976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:57.978935 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 29 22:41:58.329002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:58.328903 5610 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 29 22:41:58.329290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:58.329245 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 29 22:41:58.820085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:58.820006 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:41:58.983796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:58.983760 5610 patch_prober.go:29] interesting pod/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:41:58.985172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:58.985135 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 29 22:41:59.333141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:59.333107 5610 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 29 22:41:59.333461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:59.333427 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 29 22:41:59.937495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:59.937456 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:41:59.980415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:59.978814 5610 patch_prober.go:29] interesting pod/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:41:59.980415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:41:59.978877 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 29 22:42:00.329269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:00.329236 5610 patch_prober.go:29] interesting pod/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 29 22:42:00.329590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:00.329560 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 29 22:42:00.519979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:00.519939 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:00.701413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:00.701359 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:00.977294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:00.977156 5610 patch_prober.go:29] interesting pod/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 29 22:42:00.977294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:00.977229 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 29 22:42:01.333500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:01.333424 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:42:01.333969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:01.333945 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:42:01.334719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:01.334699 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:42:01.982921 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:01.982893 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:42:01.984199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:01.984177 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:42:02.534327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:02.534287 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:04.203750 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:04.203609 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:05.617465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:05.617425 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:07.258448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:07.258356 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:08.941085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:08.941050 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:09.383426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.383359 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t"] Oct 29 22:42:09.388572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.388551 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.418729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.418693 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e224cd\"/\"kube-root-ca.crt\"" Oct 29 22:42:09.418862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.418782 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e224cd\"/\"openshift-service-ca.crt\"" Oct 29 22:42:09.422833 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.422810 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t"] Oct 29 22:42:09.426030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.426001 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e224cd\"/\"default-dockercfg-bkjxn\"" Oct 29 22:42:09.426142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.426037 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e224cd\"/\"test-model-e224cd-kserve-self-signed-certs\"" Oct 29 22:42:09.479572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.479541 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:09.519060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.519034 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-home\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.519208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.519088 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-dshm\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.519252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.519214 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tls-certs\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.519323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.519307 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tmp-dir\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.519366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.519340 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-model-cache\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.519469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.519453 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx5rb\" (UniqueName: \"kubernetes.io/projected/3e9b513f-0968-47c2-8201-bd81a45a9e7d-kube-api-access-wx5rb\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.634974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.634822 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tmp-dir\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.635250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.635226 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-model-cache\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.635432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.635412 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5rb\" (UniqueName: \"kubernetes.io/projected/3e9b513f-0968-47c2-8201-bd81a45a9e7d-kube-api-access-wx5rb\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.635890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.635860 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-home\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.636129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.636108 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-dshm\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.636312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.636294 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tls-certs\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.645219 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.644877 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-home\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.645850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.645778 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-model-cache\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.646242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.646222 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tmp-dir\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.736846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.720335 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tls-certs\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.739117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.737382 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-dshm\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:09.796123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:09.796087 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5rb\" (UniqueName: \"kubernetes.io/projected/3e9b513f-0968-47c2-8201-bd81a45a9e7d-kube-api-access-wx5rb\") pod \"test-model-e224cd-kserve-56665b74b7-gkl6t\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:10.001780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:10.001745 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:10.320274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:10.320253 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t"] Oct 29 22:42:10.320542 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:42:10.320499 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e9b513f_0968_47c2_8201_bd81a45a9e7d.slice/crio-6dc2d7b9bcd443911e48f138d494c232d946cd1fab8d477f4596e96bbe5a3d3e WatchSource:0}: Error finding container 6dc2d7b9bcd443911e48f138d494c232d946cd1fab8d477f4596e96bbe5a3d3e: Status 404 returned error can't find the container with id 6dc2d7b9bcd443911e48f138d494c232d946cd1fab8d477f4596e96bbe5a3d3e Oct 29 22:42:10.728419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:10.722879 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:11.262160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:11.262125 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" event={"ID":"3e9b513f-0968-47c2-8201-bd81a45a9e7d","Type":"ContainerStarted","Data":"d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505"} Oct 29 22:42:11.262629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:11.262174 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" event={"ID":"3e9b513f-0968-47c2-8201-bd81a45a9e7d","Type":"ContainerStarted","Data":"6dc2d7b9bcd443911e48f138d494c232d946cd1fab8d477f4596e96bbe5a3d3e"} Oct 29 22:42:11.301425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:11.301338 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podStartSLOduration=2.301321612 podStartE2EDuration="2.301321612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:11.300080474 +0000 UTC m=+1593.781397978" watchObservedRunningTime="2026-10-29 22:42:11.301321612 +0000 UTC m=+1593.782639104" Oct 29 22:42:12.001940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:12.001904 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:12.331607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:12.331519 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:13.712427 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:42:13.712353 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:42:13.926417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:13.925765 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:15.350841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:15.350794 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:17.122898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:17.122862 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:18.121821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:18.121778 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:18.683004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:18.682909 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:20.002238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:20.002203 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:20.015369 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:20.015339 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:20.080285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:20.080251 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:20.349728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:20.349652 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:42:21.952355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:21.952324 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:22.101428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:22.100975 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:23.654158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:23.654118 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:24.181766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:24.181722 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:25.298908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:25.298866 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:26.896587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:26.896547 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:27.032991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:27.032960 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:27.508944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:27.508904 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:28.586303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:28.586266 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:30.277804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:30.277757 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:31.924147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:31.924106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:33.767286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:33.767251 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:35.707034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:35.706970 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:37.434416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:37.433912 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:38.074298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:38.074260 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:38.827402 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:38.827366 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:39.641255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:39.641217 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:39.782886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:39.782845 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:39.933104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:39.933010 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:40.306275 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:40.306241 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:41.700525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:41.700426 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:43.356414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:43.352479 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:43.711535 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:42:43.711491 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:42:44.581983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:44.581941 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:45.346421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:45.332515 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:45.491513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:45.491474 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:45.711148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:45.678338 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:46.057498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:46.057464 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:47.123171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:47.123130 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:47.275575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:47.275547 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:47.427748 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:47.426691 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt"] Oct 29 22:42:47.428179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:47.428128 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" containerID="cri-o://9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79" gracePeriod=30 Oct 29 22:42:47.439522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:47.439493 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7"] Oct 29 22:42:47.439905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:47.439875 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" containerID="cri-o://54ddf99a0440c99192d0ed50bc6577c4964f7469fbe50379f18fca0647cce1fa" gracePeriod=30 Oct 29 22:42:47.510276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:47.510238 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:47.706256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:47.706219 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:49.370452 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:49.364290 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:51.175156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:51.174911 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:52.752582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.752502 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:52.872229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.872194 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:42:52.894641 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.894606 5610 generic.go:356] "Generic (PLEG): container finished" podID="5945809b-9330-4c21-a1df-47e63d06ec59" containerID="54ddf99a0440c99192d0ed50bc6577c4964f7469fbe50379f18fca0647cce1fa" exitCode=0 Oct 29 22:42:52.895045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.895020 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" event={"ID":"5945809b-9330-4c21-a1df-47e63d06ec59","Type":"ContainerDied","Data":"54ddf99a0440c99192d0ed50bc6577c4964f7469fbe50379f18fca0647cce1fa"} Oct 29 22:42:52.897191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.897167 5610 generic.go:356] "Generic (PLEG): container finished" podID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerID="9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79" exitCode=0 Oct 29 22:42:52.897423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.897386 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" event={"ID":"175f111a-fe47-40c6-bdf3-4b94f172ccfa","Type":"ContainerDied","Data":"9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79"} Oct 29 22:42:52.897575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.897559 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" event={"ID":"175f111a-fe47-40c6-bdf3-4b94f172ccfa","Type":"ContainerDied","Data":"5b9aff9ddb7039c708ecda969856f118a1e18cad429e8469b062820f2a6b3a21"} Oct 29 22:42:52.897694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.897682 5610 scope.go:122] "RemoveContainer" containerID="9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79" Oct 29 22:42:52.897926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.897910 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt" Oct 29 22:42:52.901940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:52.901906 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:53.019177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.014387 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m"] Oct 29 22:42:53.019177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.015781 5610 cpu_manager.go:414] "containerMap: removing container" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" Oct 29 22:42:53.019177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.015802 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" Oct 29 22:42:53.019177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.016039 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" containerName="istio-proxy" Oct 29 22:42:53.026376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.025664 5610 scope.go:122] "RemoveContainer" containerID="9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79" Oct 29 22:42:53.026376 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:42:53.026080 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79\": container with ID starting with 9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79 not found: ID does not exist" containerID="9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79" Oct 29 22:42:53.026376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.026120 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79"} err="failed to get container status \"9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79\": rpc error: code = NotFound desc = could not find container \"9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79\": container with ID starting with 9177bba2da87e8117c3249e0117d237f5813195c10c80b014a3647de79956d79 not found: ID does not exist" Oct 29 22:42:53.031017 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.027109 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m"] Oct 29 22:42:53.031017 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.027221 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.065169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065131 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-data\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.065322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065230 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istiod-ca-cert\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.065322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065266 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-kube-api-access-tcggv\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-kube-api-access-tcggv\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.065463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065324 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-socket\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.065560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065540 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-certs\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.065648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065609 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-envoy\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.065720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065691 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-credential-socket\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.065777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065736 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-token\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-token\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.065854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.065839 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-podinfo\") pod \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\" (UID: \"175f111a-fe47-40c6-bdf3-4b94f172ccfa\") " Oct 29 22:42:53.067088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.067060 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-data" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.067602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.067577 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istiod-ca-cert" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:53.068489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.068460 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-credential-socket" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.068674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.068651 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-certs" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.069038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.069003 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-socket" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.072150 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.072080 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-podinfo" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:42:53.073012 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.072973 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-433712-openshift-default-dockercfg-lnlp8\"" Oct 29 22:42:53.079577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.079554 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-kube-api-access-tcggv" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "kube-api-access-tcggv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:53.080578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.080552 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-envoy" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.082158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.082136 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-token" pod "175f111a-fe47-40c6-bdf3-4b94f172ccfa" (UID: "175f111a-fe47-40c6-bdf3-4b94f172ccfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:53.083042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.083020 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:42:53.147556 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.146429 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8"] Oct 29 22:42:53.148358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.148326 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" Oct 29 22:42:53.148358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.148360 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" Oct 29 22:42:53.148611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.148589 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" containerName="istio-proxy" Oct 29 22:42:53.152095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.152077 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.158584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.158557 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr"] Oct 29 22:42:53.158827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.158792 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" containerID="cri-o://78f3d83c12cdefd19e2e1c78cb4b374cc8fc2f20d254817de5f825b4b1281057" gracePeriod=30 Oct 29 22:42:53.172573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.170234 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.172573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.170283 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.172573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.170366 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.172573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.170434 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.172573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.170600 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8l4r\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-kube-api-access-v8l4r\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.172573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.170752 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.172573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.170814 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.172573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.170864 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173227 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173415 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173438 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173453 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173467 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173481 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173494 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173506 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173520 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175f111a-fe47-40c6-bdf3-4b94f172ccfa-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.173554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.173533 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcggv\" (UniqueName: \"kubernetes.io/projected/175f111a-fe47-40c6-bdf3-4b94f172ccfa-kube-api-access-tcggv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.177535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.177339 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj"] Oct 29 22:42:53.177974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.177928 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" containerID="cri-o://c7bc8f6340b104e658a57608a5a3b3e36d5a638ecc161838e157468edf5506a2" gracePeriod=30 Oct 29 22:42:53.185101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.185083 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh"] Oct 29 22:42:53.185561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.185534 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" containerID="cri-o://1e08596386a34a03fb9c83c21c8992be14b5171e309b986d7042c0ba533f1303" gracePeriod=30 Oct 29 22:42:53.192322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.192284 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8"] Oct 29 22:42:53.208247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.202201 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m"] Oct 29 22:42:53.208478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.208461 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr"] Oct 29 22:42:53.208842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.208816 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" containerID="cri-o://dc5c4c7f97ce7a1608be64e9f0b1c9ccac889a27fc2b83f439e97d1d64bb2e6e" gracePeriod=30 Oct 29 22:42:53.209913 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.208767 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" containerID="cri-o://97ed0bd2e794f6cda1c613130eead2d3b57ede2a38487af99d86bfe7c6d93b94" gracePeriod=30 Oct 29 22:42:53.265217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.258372 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:53.380468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.364085 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-kube-api-access-m5lpd\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-kube-api-access-m5lpd\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.380468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.364146 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-data\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.380468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.364267 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5945809b-9330-4c21-a1df-47e63d06ec59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5945809b-9330-4c21-a1df-47e63d06ec59-istiod-ca-cert\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.380468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.364309 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-credential-socket\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.391689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.380913 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5945809b-9330-4c21-a1df-47e63d06ec59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5945809b-9330-4c21-a1df-47e63d06ec59-istio-podinfo\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.391689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.380986 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-certs\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.391689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.381137 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-envoy\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.391689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.381177 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-socket\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.391689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.381303 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-istio-token\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-istio-token\") pod \"5945809b-9330-4c21-a1df-47e63d06ec59\" (UID: \"5945809b-9330-4c21-a1df-47e63d06ec59\") " Oct 29 22:42:53.422520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.422486 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.422802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.422778 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e132f0f-c890-4055-8967-bbaf48db60d2-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.422957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.422942 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.428207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.428135 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvfn4\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-kube-api-access-gvfn4\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.428510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.428476 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.434988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.434943 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.435268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.435243 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.463541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.463496 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.463825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.463805 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.463957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.463942 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.464119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.464102 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.466308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.466281 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.466477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.440973 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.466616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.441350 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.466982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.466958 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.469468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.467729 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.469683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.469662 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.482975 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8l4r\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-kube-api-access-v8l4r\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.483083 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.483179 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.483242 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.483279 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.483948 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.428708 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-data" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.435560 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5945809b-9330-4c21-a1df-47e63d06ec59-istiod-ca-cert" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.464264 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-credential-socket" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.484722 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-socket" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.497658 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.484943 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-certs" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.557416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.549719 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.570470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.563563 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-kube-api-access-m5lpd" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "kube-api-access-m5lpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:53.573037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.573001 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.573632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.573610 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8l4r\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-kube-api-access-v8l4r\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603039 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603098 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e132f0f-c890-4055-8967-bbaf48db60d2-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603151 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603174 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvfn4\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-kube-api-access-gvfn4\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603275 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603317 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603461 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603521 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603628 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603777 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603793 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5lpd\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-kube-api-access-m5lpd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603808 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603822 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5945809b-9330-4c21-a1df-47e63d06ec59-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603834 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.603847 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.604225 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.604500 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.605388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.605089 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e132f0f-c890-4055-8967-bbaf48db60d2-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.618467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.607753 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.618467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.608243 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.618467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.610537 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-envoy" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:53.618467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.610676 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-istio-token" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:53.623335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.623148 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-zsz7m\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.624782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.624739 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.632054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.631973 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvfn4\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-kube-api-access-gvfn4\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.632675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.632424 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.633452 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.633291 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-z47b8\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.633752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.633674 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5945809b-9330-4c21-a1df-47e63d06ec59-istio-podinfo" pod "5945809b-9330-4c21-a1df-47e63d06ec59" (UID: "5945809b-9330-4c21-a1df-47e63d06ec59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:42:53.687417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.682067 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:53.706336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.706297 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5945809b-9330-4c21-a1df-47e63d06ec59-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.706336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.706332 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5945809b-9330-4c21-a1df-47e63d06ec59-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.706577 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.706347 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5945809b-9330-4c21-a1df-47e63d06ec59-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:53.724888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.724852 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt"] Oct 29 22:42:53.730199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.730171 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-2hdmt"] Oct 29 22:42:53.792378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.792119 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:53.924757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.920531 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" Oct 29 22:42:53.924757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.922405 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7" event={"ID":"5945809b-9330-4c21-a1df-47e63d06ec59","Type":"ContainerDied","Data":"4c9ee2169c58c17429ecd0429b63964c03a791c2096576068dbd31534946ecb5"} Oct 29 22:42:53.924757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.922486 5610 scope.go:122] "RemoveContainer" containerID="54ddf99a0440c99192d0ed50bc6577c4964f7469fbe50379f18fca0647cce1fa" Oct 29 22:42:53.981774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.981737 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7"] Oct 29 22:42:53.988082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:53.988044 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0ab33e-openshift-default-65bc44c749-72qh7"] Oct 29 22:42:54.022005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.021962 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:42:54.024451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.022624 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:42:54.024451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.022730 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:42:54.032731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.032621 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m"] Oct 29 22:42:54.144200 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.144169 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8"] Oct 29 22:42:54.145016 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:42:54.144985 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e132f0f_c890_4055_8967_bbaf48db60d2.slice/crio-4ee916ebdda4c842c6ac4b100bacc744be7e8dcdc19121c422883af3879d59df WatchSource:0}: Error finding container 4ee916ebdda4c842c6ac4b100bacc744be7e8dcdc19121c422883af3879d59df: Status 404 returned error can't find the container with id 4ee916ebdda4c842c6ac4b100bacc744be7e8dcdc19121c422883af3879d59df Oct 29 22:42:54.149537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.149498 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:42:54.149644 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.149588 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:42:54.149644 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.149625 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:42:54.769042 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.768945 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:54.934194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.934160 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" event={"ID":"cfe7e5db-25cf-4299-a4ce-1a7e41598d66","Type":"ContainerStarted","Data":"aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb"} Oct 29 22:42:54.934716 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.934229 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" event={"ID":"cfe7e5db-25cf-4299-a4ce-1a7e41598d66","Type":"ContainerStarted","Data":"b5de50ae30f20d651f77362719b8e6aba93ec9fa38ad742a7952c4c39a88c03f"} Oct 29 22:42:54.935985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.935953 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" event={"ID":"8e132f0f-c890-4055-8967-bbaf48db60d2","Type":"ContainerStarted","Data":"67e91b0aa190fa5515a64739caffeb6b051e2c411aac584cbffd649dfcc2f783"} Oct 29 22:42:54.936202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.936168 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" event={"ID":"8e132f0f-c890-4055-8967-bbaf48db60d2","Type":"ContainerStarted","Data":"4ee916ebdda4c842c6ac4b100bacc744be7e8dcdc19121c422883af3879d59df"} Oct 29 22:42:54.981811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:54.981763 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" podStartSLOduration=63.98174959 podStartE2EDuration="1m3.98174959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:54.978149923 +0000 UTC m=+1637.459467416" watchObservedRunningTime="2026-10-29 22:42:54.98174959 +0000 UTC m=+1637.463067082" Oct 29 22:42:55.022019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.021899 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" podStartSLOduration=64.021880789 podStartE2EDuration="1m4.021880789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:55.018683668 +0000 UTC m=+1637.500001184" watchObservedRunningTime="2026-10-29 22:42:55.021880789 +0000 UTC m=+1637.503198281" Oct 29 22:42:55.640581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.640536 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175f111a-fe47-40c6-bdf3-4b94f172ccfa" path="/var/lib/kubelet/pods/175f111a-fe47-40c6-bdf3-4b94f172ccfa/volumes" Oct 29 22:42:55.641261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.641233 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5945809b-9330-4c21-a1df-47e63d06ec59" path="/var/lib/kubelet/pods/5945809b-9330-4c21-a1df-47e63d06ec59/volumes" Oct 29 22:42:55.689376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.682813 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:42:55.703416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.694670 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:42:55.703416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.694736 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:42:55.803376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.798763 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:42:55.808488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.805532 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-z47b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 22:42:55.808488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:55.805592 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 22:42:56.685424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:56.684782 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:42:56.685424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:56.684843 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:42:56.776954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:56.776880 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:56.807092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:56.806527 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-z47b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 22:42:56.807092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:56.806639 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 22:42:57.044959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.043360 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:42:57.044959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.043444 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:42:57.389911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.389658 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:42:57.389911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.389759 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:42:57.693475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.692418 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:42:57.693475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.692480 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:42:57.801418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.801169 5610 patch_prober.go:29] interesting pod/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:42:57.801418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.801235 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:42:57.809419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.802363 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-z47b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 22:42:57.809419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:57.802424 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 22:42:58.450885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:58.450148 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r"] Oct 29 22:42:58.892601 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:58.839216 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:42:58.892601 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:58.839288 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:42:58.916708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:58.905689 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-z47b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 22:42:58.916708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:58.905755 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 22:42:58.930331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:58.929574 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:59.643062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.640371 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:42:59.728749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.716557 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:42:59.728749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.716645 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:42:59.751520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.751213 5610 generic.go:356] "Generic (PLEG): container finished" podID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerID="dc5c4c7f97ce7a1608be64e9f0b1c9ccac889a27fc2b83f439e97d1d64bb2e6e" exitCode=0 Oct 29 22:42:59.756982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.756947 5610 generic.go:356] "Generic (PLEG): container finished" podID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerID="78f3d83c12cdefd19e2e1c78cb4b374cc8fc2f20d254817de5f825b4b1281057" exitCode=0 Oct 29 22:42:59.788552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.784786 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" event={"ID":"e5186bd9-b725-4a0d-b5a7-e8adefc36228","Type":"ContainerDied","Data":"dc5c4c7f97ce7a1608be64e9f0b1c9ccac889a27fc2b83f439e97d1d64bb2e6e"} Oct 29 22:42:59.788552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.784857 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" event={"ID":"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6","Type":"ContainerDied","Data":"78f3d83c12cdefd19e2e1c78cb4b374cc8fc2f20d254817de5f825b4b1281057"} Oct 29 22:42:59.841415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.840930 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-z47b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 22:42:59.841415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.841005 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 22:42:59.844487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.843980 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:42:59.850060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.848493 5610 generic.go:356] "Generic (PLEG): container finished" podID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerID="c7bc8f6340b104e658a57608a5a3b3e36d5a638ecc161838e157468edf5506a2" exitCode=0 Oct 29 22:42:59.850060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.848616 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" event={"ID":"786348c4-3d9b-4d2b-8094-f9f395568a8a","Type":"ContainerDied","Data":"c7bc8f6340b104e658a57608a5a3b3e36d5a638ecc161838e157468edf5506a2"} Oct 29 22:42:59.852084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.851726 5610 generic.go:356] "Generic (PLEG): container finished" podID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerID="97ed0bd2e794f6cda1c613130eead2d3b57ede2a38487af99d86bfe7c6d93b94" exitCode=0 Oct 29 22:42:59.852084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.851859 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" event={"ID":"1106bfcd-df18-4806-9286-c5b96aff17d5","Type":"ContainerDied","Data":"97ed0bd2e794f6cda1c613130eead2d3b57ede2a38487af99d86bfe7c6d93b94"} Oct 29 22:42:59.853136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.852673 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:42:59.855382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.855315 5610 generic.go:356] "Generic (PLEG): container finished" podID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerID="1e08596386a34a03fb9c83c21c8992be14b5171e309b986d7042c0ba533f1303" exitCode=0 Oct 29 22:42:59.855382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.855379 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" event={"ID":"d3e57308-cd83-4966-9847-e8851bd7ce38","Type":"ContainerDied","Data":"1e08596386a34a03fb9c83c21c8992be14b5171e309b986d7042c0ba533f1303"} Oct 29 22:42:59.865860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.865835 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-kube-api-access-jdq8g\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-kube-api-access-jdq8g\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.866407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.866370 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-podinfo\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.867788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.867770 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istiod-ca-cert\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.868291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.868275 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-credential-socket\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.869971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.869952 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-token\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-token\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.870228 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.870210 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-data\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.872027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.872008 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-certs\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.874441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.868219 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istiod-ca-cert" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:42:59.874582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.868793 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-credential-socket" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:59.874683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.870630 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-data" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:59.874774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.872439 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-certs" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:59.874922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.874905 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-socket\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.875323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.875305 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-envoy\") pod \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\" (UID: \"e5186bd9-b725-4a0d-b5a7-e8adefc36228\") " Oct 29 22:42:59.876324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.875527 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-socket" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:59.876942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.876922 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:42:59.877561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.877541 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:59.877786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.877751 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:59.877954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.877937 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:59.878106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.878091 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:59.878276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.878219 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:42:59.889070 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.889026 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-token" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:59.889311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.889272 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-envoy" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:42:59.889573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.889543 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-kube-api-access-jdq8g" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "kube-api-access-jdq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:42:59.889773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.889737 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-podinfo" pod "e5186bd9-b725-4a0d-b5a7-e8adefc36228" (UID: "e5186bd9-b725-4a0d-b5a7-e8adefc36228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:42:59.956021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.955954 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2"] Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957136 5610 cpu_manager.go:414] "containerMap: removing container" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957158 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957177 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957183 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957235 5610 cpu_manager.go:414] "containerMap: removing container" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957243 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957381 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" containerName="istio-proxy" Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957404 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" containerName="istio-proxy" Oct 29 22:42:59.989727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:42:59.957430 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" containerName="istio-proxy" Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002489 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-socket\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002587 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-kube-api-access-bf747\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-kube-api-access-bf747\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002657 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-token\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-token\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002721 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-kube-api-access-x8j4q\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-kube-api-access-x8j4q\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002757 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-credential-socket\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002838 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-envoy\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002870 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-token\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-token\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002908 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-podinfo\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.002965 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-podinfo\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003089 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istiod-ca-cert\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003163 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1106bfcd-df18-4806-9286-c5b96aff17d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106bfcd-df18-4806-9286-c5b96aff17d5-istiod-ca-cert\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003195 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-certs\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003276 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-data\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003319 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-data\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003365 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-socket\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003419 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-credential-socket\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003449 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-envoy\") pod \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\" (UID: \"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.003556 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-certs\") pod \"1106bfcd-df18-4806-9286-c5b96aff17d5\" (UID: \"1106bfcd-df18-4806-9286-c5b96aff17d5\") " Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.041065 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.042109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.042056 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-credential-socket" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.043252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.042951 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-data" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.043410 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.043348 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istiod-ca-cert" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:00.046962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.046726 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-certs" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.047121 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.047044 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1106bfcd-df18-4806-9286-c5b96aff17d5-istiod-ca-cert" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:00.050118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.049942 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-data" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.050504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.050456 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-credential-socket" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.050827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.050802 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-certs" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.051002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.050978 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.055588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.051979 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-socket" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.055588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.054144 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-envoy" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.055588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.054240 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdq8g\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-kube-api-access-jdq8g\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.055770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.055694 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.055770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.055734 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5186bd9-b725-4a0d-b5a7-e8adefc36228-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.064458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.056150 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-socket" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.064458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.056898 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d478b36e-serving-cert\"" Oct 29 22:43:00.064458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.057075 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-podinfo" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:00.093479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.069276 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-kube-api-access-x8j4q" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "kube-api-access-x8j4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:00.120817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.120716 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:43:00.130464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.129967 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2"] Oct 29 22:43:00.267419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.266364 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-token\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-token\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.267419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.266882 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvgf\" (UniqueName: \"kubernetes.io/projected/030c3c93-fe58-45ce-a46e-66c36a44b801-kube-api-access-ssvgf\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.267419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.267248 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-maas-api-tls\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.413505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.413457 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-podinfo" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438050 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030c3c93-fe58-45ce-a46e-66c36a44b801-openshift-service-ca\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438096 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-metrics-certs\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438282 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438299 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106bfcd-df18-4806-9286-c5b96aff17d5-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438313 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438326 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438340 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438355 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438369 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438383 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438417 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438430 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438444 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8j4q\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-kube-api-access-x8j4q\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438462 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.448033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.438478 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.479639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.479571 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-envoy" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.515611 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-kube-api-access-bf747" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "kube-api-access-bf747". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.540287 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-socket\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.540378 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-data\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.540438 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-credential-socket\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.540493 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-envoy\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.541282 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-certs\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.541338 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-podinfo\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.541372 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e57308-cd83-4966-9847-e8851bd7ce38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e57308-cd83-4966-9847-e8851bd7ce38-istiod-ca-cert\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.541474 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.541767 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030c3c93-fe58-45ce-a46e-66c36a44b801-openshift-service-ca\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.541800 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-metrics-certs\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.541981 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvgf\" (UniqueName: \"kubernetes.io/projected/030c3c93-fe58-45ce-a46e-66c36a44b801-kube-api-access-ssvgf\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.542260 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-maas-api-tls\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.542474 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf747\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-kube-api-access-bf747\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.542494 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.542511 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.547010 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-certs" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.547827 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-socket" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.548048 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-data" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.548248 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-credential-socket" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.552871 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030c3c93-fe58-45ce-a46e-66c36a44b801-openshift-service-ca\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.595487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.553238 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e57308-cd83-4966-9847-e8851bd7ce38-istiod-ca-cert" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:00.642427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.636516 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-envoy" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.642427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.641293 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-podinfo" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:00.642427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.641501 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-metrics-certs\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.642427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.641933 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-maas-api-tls\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.642427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.642198 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvgf\" (UniqueName: \"kubernetes.io/projected/030c3c93-fe58-45ce-a46e-66c36a44b801-kube-api-access-ssvgf\") pod \"maas-api-e2e-mt-d478b36e-75bc765888-t8tg2\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.643697 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "kube-api-access-hxdns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644079 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns\") pod \"d3e57308-cd83-4966-9847-e8851bd7ce38\" (UID: \"d3e57308-cd83-4966-9847-e8851bd7ce38\") " Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644793 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644813 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644827 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e57308-cd83-4966-9847-e8851bd7ce38-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644841 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644855 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644868 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644882 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:43:00.644956 5610 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/d3e57308-cd83-4966-9847-e8851bd7ce38/volumes/kubernetes.io~projected/kube-api-access-hxdns Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.644998 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "kube-api-access-hxdns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.647035 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-token" pod "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" (UID: "20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:00.651563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.650821 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-token" pod "1106bfcd-df18-4806-9286-c5b96aff17d5" (UID: "1106bfcd-df18-4806-9286-c5b96aff17d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:00.652635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.652578 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-token" pod "d3e57308-cd83-4966-9847-e8851bd7ce38" (UID: "d3e57308-cd83-4966-9847-e8851bd7ce38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:00.655981 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.655564 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj"] Oct 29 22:43:00.677143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.657260 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" Oct 29 22:43:00.677143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.657306 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" Oct 29 22:43:00.677143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.657717 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" containerName="istio-proxy" Oct 29 22:43:00.693575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.683716 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.731888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.695886 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 29 22:43:00.731888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.695939 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 29 22:43:00.731888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.724465 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj"] Oct 29 22:43:00.745946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.745881 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk"] Oct 29 22:43:00.749422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.748632 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.749422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.748661 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106bfcd-df18-4806-9286-c5b96aff17d5-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.749422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.748675 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.749422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.748688 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdns\" (UniqueName: \"kubernetes.io/projected/d3e57308-cd83-4966-9847-e8851bd7ce38-kube-api-access-hxdns\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:00.755669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.754903 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.755830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.755809 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:43:00.756593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.756568 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk"] Oct 29 22:43:00.819171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.817904 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-z47b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 29 22:43:00.819171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.817963 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.834077 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850225 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850268 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850293 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcbl\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-kube-api-access-4fcbl\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850445 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850502 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8913333-9e24-419a-998c-6e8c604e994e-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850695 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850720 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850773 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.879471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.850797 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8913333-9e24-419a-998c-6e8c604e994e-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.933494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.933456 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp"] Oct 29 22:43:00.935105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.935077 5610 cpu_manager.go:414] "containerMap: removing container" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" Oct 29 22:43:00.935269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.935255 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" Oct 29 22:43:00.935635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.935617 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" containerName="istio-proxy" Oct 29 22:43:00.950643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.950615 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp"] Oct 29 22:43:00.950793 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.950731 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:00.951789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.951762 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-kube-api-access-8hfz9\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-kube-api-access-8hfz9\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.951938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.951923 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-token\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-token\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.952010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.951950 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-credential-socket\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.952010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.951973 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-podinfo\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.952142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952022 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-certs\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.952142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952064 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-socket\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.952142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952126 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786348c4-3d9b-4d2b-8094-f9f395568a8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786348c4-3d9b-4d2b-8094-f9f395568a8a-istiod-ca-cert\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.952296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952192 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.952296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952217 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-data\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:00.952434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952356 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.952434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952382 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.952582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952450 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.952582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952479 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.952582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952508 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcbl\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-kube-api-access-4fcbl\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.952582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952540 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.952781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952609 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.952781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952649 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.952781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952680 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8913333-9e24-419a-998c-6e8c604e994e-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.952781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952729 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.952993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952786 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce88088-ea36-4392-8466-b74c56c3111d-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.952993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.952950 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-kube-api-access-w5sdf\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.953100 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953013 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.953100 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953041 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.953215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953107 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.953215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953144 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.953215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953172 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8913333-9e24-419a-998c-6e8c604e994e-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.953215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953196 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:00.953476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953366 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce88088-ea36-4392-8466-b74c56c3111d-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:00.953476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953460 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-socket" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.953818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953782 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786348c4-3d9b-4d2b-8094-f9f395568a8a-istiod-ca-cert" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:00.953907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.953889 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-certs" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.957056 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.957030 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-data" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.957497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.957421 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-credential-socket" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:00.957979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:00.957954 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.008416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.006513 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.008416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.006817 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.008416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.008004 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8913333-9e24-419a-998c-6e8c604e994e-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.009796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.009755 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" event={"ID":"e5186bd9-b725-4a0d-b5a7-e8adefc36228","Type":"ContainerDied","Data":"91395f05b1a0032600d5fa46ed0ebea712d67b19581aad27fb4452ee7a714b8d"} Oct 29 22:43:01.010017 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.009987 5610 scope.go:122] "RemoveContainer" containerID="dc5c4c7f97ce7a1608be64e9f0b1c9ccac889a27fc2b83f439e97d1d64bb2e6e" Oct 29 22:43:01.045231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.010199 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr" Oct 29 22:43:01.048761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.048709 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.048941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.017833 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" Oct 29 22:43:01.051114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.051084 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-kube-api-access-8hfz9" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "kube-api-access-8hfz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:01.051114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.043379 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" Oct 29 22:43:01.056628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.055878 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcbl\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-kube-api-access-4fcbl\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.056628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.055879 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.056628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.017767 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr" event={"ID":"20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6","Type":"ContainerDied","Data":"34a01be6184a982d1c47350329baa6d78f7063e913d91f1cdb8dd2ac4a7f3ac8"} Oct 29 22:43:01.056628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.055968 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" event={"ID":"786348c4-3d9b-4d2b-8094-f9f395568a8a","Type":"ContainerDied","Data":"cb87e6c8a3ad39c189456a6e271ea3464697535018f36453c0a82d7b2a48ea97"} Oct 29 22:43:01.056628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.055999 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m" event={"ID":"1106bfcd-df18-4806-9286-c5b96aff17d5","Type":"ContainerDied","Data":"949f7a907c0718ab39439a797bbe2927b2ca38f9ed8dedeadc08462a20afbb4b"} Oct 29 22:43:01.056628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.055999 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-token" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:01.056628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.056065 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-podinfo" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:01.056974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.056770 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8913333-9e24-419a-998c-6e8c604e994e-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-8wbcj\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.057210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.057088 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:01.057210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.021090 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.057620 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy\") pod \"786348c4-3d9b-4d2b-8094-f9f395568a8a\" (UID: \"786348c4-3d9b-4d2b-8094-f9f395568a8a\") " Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.057927 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.057968 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce88088-ea36-4392-8466-b74c56c3111d-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.057996 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058032 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058061 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058107 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058170 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058204 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0301254-4c8b-47b7-9011-c896efee6212-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058238 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058290 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce88088-ea36-4392-8466-b74c56c3111d-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058339 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058444 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0301254-4c8b-47b7-9011-c896efee6212-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058496 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-kube-api-access-w5sdf\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058527 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058553 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d69d\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-kube-api-access-9d69d\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058625 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058661 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058688 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058815 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058830 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058844 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058856 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058869 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058881 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/786348c4-3d9b-4d2b-8094-f9f395568a8a-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058895 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.058909 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hfz9\" (UniqueName: \"kubernetes.io/projected/786348c4-3d9b-4d2b-8094-f9f395568a8a-kube-api-access-8hfz9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:43:01.059006 5610 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/786348c4-3d9b-4d2b-8094-f9f395568a8a/volumes/kubernetes.io~empty-dir/istio-envoy Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.059017 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy" pod "786348c4-3d9b-4d2b-8094-f9f395568a8a" (UID: "786348c4-3d9b-4d2b-8094-f9f395568a8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.065654 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.066150 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.066425 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.071414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.066671 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.133504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.133385 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce88088-ea36-4392-8466-b74c56c3111d-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.151809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.150643 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.152413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.152364 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce88088-ea36-4392-8466-b74c56c3111d-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.154664 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.154635 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" event={"ID":"d3e57308-cd83-4966-9847-e8851bd7ce38","Type":"ContainerDied","Data":"e98b16876450799c31dcae39a67fe281e7c1501b96c7a7ed9fc93f9c9729b30c"} Oct 29 22:43:01.154849 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.154821 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:01.155593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.155567 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh" Oct 29 22:43:01.158621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.156657 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:01.188272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.187973 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0301254-4c8b-47b7-9011-c896efee6212-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.188272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.188051 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.188272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.188094 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d69d\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-kube-api-access-9d69d\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.188272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.188199 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.188272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.188234 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.196082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.188369 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.290287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.290188 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.291604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.291578 5610 scope.go:122] "RemoveContainer" containerID="78f3d83c12cdefd19e2e1c78cb4b374cc8fc2f20d254817de5f825b4b1281057" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:01.388403 5610 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5186bd9_b725_4a0d_b5a7_e8adefc36228.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.389902 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.392330 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-credential-socket\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.392636 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-certs\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.392694 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.392900 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0301254-4c8b-47b7-9011-c896efee6212-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.393015 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.393274 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/786348c4-3d9b-4d2b-8094-f9f395568a8a-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:01.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.393587 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-data\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.421509 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.413250 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-kube-api-access-w5sdf\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.459671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.446675 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-envoy\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.459671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.448741 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0301254-4c8b-47b7-9011-c896efee6212-istiod-ca-cert\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.459671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.450416 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0301254-4c8b-47b7-9011-c896efee6212-istio-podinfo\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.459671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.454686 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d69d\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-kube-api-access-9d69d\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-ddrwp\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.459671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.455634 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-istio-token\") pod \"e2e-api-b-433712-openshift-default-594dc9f747-lr5vk\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.532905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.532834 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m"] Oct 29 22:43:01.542734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.541045 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-cw88m"] Oct 29 22:43:01.556870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.556829 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr"] Oct 29 22:43:01.557091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.557077 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tclfr"] Oct 29 22:43:01.558370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.558343 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:01.649923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.639556 5610 scope.go:122] "RemoveContainer" containerID="c7bc8f6340b104e658a57608a5a3b3e36d5a638ecc161838e157468edf5506a2" Oct 29 22:43:01.748485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.733378 5610 scope.go:122] "RemoveContainer" containerID="97ed0bd2e794f6cda1c613130eead2d3b57ede2a38487af99d86bfe7c6d93b94" Oct 29 22:43:01.748485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.742454 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1106bfcd-df18-4806-9286-c5b96aff17d5" path="/var/lib/kubelet/pods/1106bfcd-df18-4806-9286-c5b96aff17d5/volumes" Oct 29 22:43:01.748485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.743285 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5186bd9-b725-4a0d-b5a7-e8adefc36228" path="/var/lib/kubelet/pods/e5186bd9-b725-4a0d-b5a7-e8adefc36228/volumes" Oct 29 22:43:01.748485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.743777 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr"] Oct 29 22:43:01.748485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.743816 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-tkndr"] Oct 29 22:43:01.748485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.743843 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2"] Oct 29 22:43:01.753940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.752386 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh"] Oct 29 22:43:01.758196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.757504 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:01.758196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.757976 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-dq9jh"] Oct 29 22:43:01.790694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.790670 5610 scope.go:122] "RemoveContainer" containerID="1e08596386a34a03fb9c83c21c8992be14b5171e309b986d7042c0ba533f1303" Oct 29 22:43:01.811484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.811420 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj"] Oct 29 22:43:01.817811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:01.817774 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f76fb5-openshift-default-7755f59d9c-r5hmj"] Oct 29 22:43:02.027712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.027683 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:43:02.029581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.029559 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:43:02.031624 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.031606 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:43:02.123455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.116528 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:43:02.123455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.119898 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:43:02.125507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.125466 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:43:02.125689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.125668 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj"] Oct 29 22:43:02.140905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.140875 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.140986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.140956 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.141137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.141003 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.174254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.173712 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp"] Oct 29 22:43:02.227839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.227752 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" event={"ID":"030c3c93-fe58-45ce-a46e-66c36a44b801","Type":"ContainerStarted","Data":"00d980a40c87cdcaef424ece9b632974a71489ad095f523092ef6ab84f5e2ca5"} Oct 29 22:43:02.233879 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.233850 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" event={"ID":"b8913333-9e24-419a-998c-6e8c604e994e","Type":"ContainerStarted","Data":"308c76173325a70efbd4cb7bee3eb55154a7247b1badd55b9256799639cd08bc"} Oct 29 22:43:02.245668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.238342 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.245668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.238444 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.245668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.238492 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.245668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.241284 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.245668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.241362 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.245668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.241423 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:02.250069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.250046 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk"] Oct 29 22:43:02.613506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.613465 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:02.774971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:02.774908 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:03.370109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.368751 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" event={"ID":"b8913333-9e24-419a-998c-6e8c604e994e","Type":"ContainerStarted","Data":"9e6edcae2aca1bea380218593365dae7e114fc6de6e09e28347de4fbcd16e841"} Oct 29 22:43:03.453442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.452108 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" podUID="b5ea960c-2e98-4332-bb5a-6a6b38f9d63c" containerName="payload-pre-processing" containerID="cri-o://7408d96c74f6ac4649c68ce0ed3d012ae2a6c8b57314ed3abbb9d38f45156b6d" gracePeriod=25 Oct 29 22:43:03.487420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.465354 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" event={"ID":"e0301254-4c8b-47b7-9011-c896efee6212","Type":"ContainerStarted","Data":"a946d81c76a0acf46d72e19829074c17c8acd0d26f986b7b84d299d6da1ce00c"} Oct 29 22:43:03.487420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.465442 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" event={"ID":"e0301254-4c8b-47b7-9011-c896efee6212","Type":"ContainerStarted","Data":"d5c62f3a3beda288ac9ed0bda43c1dbab4d95159e51a7560966c1e5d7d08aee1"} Oct 29 22:43:03.562086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.550321 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" event={"ID":"030c3c93-fe58-45ce-a46e-66c36a44b801","Type":"ContainerStarted","Data":"a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5"} Oct 29 22:43:03.562086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.551605 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:03.593273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.569779 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podStartSLOduration=57.56975782 podStartE2EDuration="57.56975782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:03.452940313 +0000 UTC m=+1645.934257811" watchObservedRunningTime="2026-10-29 22:43:03.56975782 +0000 UTC m=+1646.051075315" Oct 29 22:43:03.597054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.572002 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" event={"ID":"fce88088-ea36-4392-8466-b74c56c3111d","Type":"ContainerStarted","Data":"294f76ea4243d3f3e262919bc31fddeec97bb235c1ab7b0d1be1e440f989c341"} Oct 29 22:43:03.597054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.594830 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:03.597054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.594861 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" event={"ID":"fce88088-ea36-4392-8466-b74c56c3111d","Type":"ContainerStarted","Data":"5e9be5ce0d9f175b3c3bcd05508d2f67b79cfe12ea1bcc6247f8a3847a825412"} Oct 29 22:43:03.597054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.578645 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:03.597054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.594931 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:03.606525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.602575 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podStartSLOduration=117.602555873 podStartE2EDuration="1m57.602555873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:03.544087766 +0000 UTC m=+1646.025405260" watchObservedRunningTime="2026-10-29 22:43:03.602555873 +0000 UTC m=+1646.083873367" Oct 29 22:43:03.744448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.744369 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" podStartSLOduration=63.359454343 podStartE2EDuration="1m3.744348637s" totalImagesPullingTime="384.894294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:03.595619545 +0000 UTC m=+1646.076937039" watchObservedRunningTime="2026-10-29 22:43:03.744348637 +0000 UTC m=+1646.225666130" Oct 29 22:43:03.808442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.791115 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:03.808442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.791193 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:03.866416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:03.860355 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:04.235114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.222063 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:04.235114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.222139 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:04.257417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.255678 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podStartSLOduration=118.25565717 podStartE2EDuration="1m58.25565717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:03.649261114 +0000 UTC m=+1646.130578588" watchObservedRunningTime="2026-10-29 22:43:04.25565717 +0000 UTC m=+1646.736974663" Oct 29 22:43:04.423332 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.422990 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:04.428080 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.428047 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:04.475448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.475411 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6" path="/var/lib/kubelet/pods/20d18ee4-ca25-4cf0-b8b0-33d2ff273ea6/volumes" Oct 29 22:43:04.583418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.572343 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:04.583736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.583686 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:04.625416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.609474 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786348c4-3d9b-4d2b-8094-f9f395568a8a" path="/var/lib/kubelet/pods/786348c4-3d9b-4d2b-8094-f9f395568a8a/volumes" Oct 29 22:43:04.640420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.637515 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e57308-cd83-4966-9847-e8851bd7ce38" path="/var/lib/kubelet/pods/d3e57308-cd83-4966-9847-e8851bd7ce38/volumes" Oct 29 22:43:04.640420 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:04.638063 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.006s" Oct 29 22:43:04.640420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.638110 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:04.640420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.638336 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:04.787494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.786297 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:04.787494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.786363 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:04.852438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:04.837757 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": context deadline exceeded" Oct 29 22:43:05.171610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.158758 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:05.191802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.191762 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:05.192231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.192205 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:05.218419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.196470 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:05.219069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.203292 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:05.245766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.245464 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh"] Oct 29 22:43:05.272427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.266429 5610 generic.go:356] "Generic (PLEG): container finished" podID="b5ea960c-2e98-4332-bb5a-6a6b38f9d63c" containerID="7408d96c74f6ac4649c68ce0ed3d012ae2a6c8b57314ed3abbb9d38f45156b6d" exitCode=0 Oct 29 22:43:05.452411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.449685 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-rdht5" podUID="a4f450ff-4b49-4058-b373-d66951b19315" containerName="proxy" probeResult="failure" output="Get \"https://10.134.0.33:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:05.452411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.449941 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:05.454134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.454093 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:05.548415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.543821 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg"] Oct 29 22:43:05.558415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.557634 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:05.611968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.611929 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:43:05.612194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.612169 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" event={"ID":"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c","Type":"ContainerDied","Data":"7408d96c74f6ac4649c68ce0ed3d012ae2a6c8b57314ed3abbb9d38f45156b6d"} Oct 29 22:43:05.629460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.613385 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:05.837234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.837191 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:05.837413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.837259 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:05.915365 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.915331 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh"] Oct 29 22:43:05.915727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.915712 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg"] Oct 29 22:43:05.968287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.968254 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-139eb7\"" Oct 29 22:43:05.996223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.973296 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:05.996223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.994114 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:05.996223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:05.974931 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-139eb7-dockercfg-kf5lr\"" Oct 29 22:43:06.171469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.168024 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:06.181448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.171715 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:06.238485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.227433 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:06.539415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.538576 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 29 22:43:06.605893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.604196 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:06.627463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.615472 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:06.804529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.780132 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:06.816844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.805458 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:06.816844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.805515 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:06.850417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.818810 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92999ac-9100-4f7e-9010-f9f720d5e4a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh\" (UID: \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:06.882571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.856677 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnvs\" (UniqueName: \"kubernetes.io/projected/e92999ac-9100-4f7e-9010-f9f720d5e4a7-kube-api-access-clnvs\") pod \"payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh\" (UID: \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:06.882571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.856872 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f5dc-2049-48f7-aef2-55d5e5956f5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg\" (UID: \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:06.882571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:06.856915 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn79b\" (UniqueName: \"kubernetes.io/projected/1530f5dc-2049-48f7-aef2-55d5e5956f5f-kube-api-access-bn79b\") pod \"payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg\" (UID: \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:07.074533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.017578 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": context deadline exceeded" Oct 29 22:43:07.074533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.019092 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:07.074533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.021202 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:07.074533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.021737 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:43:07.101873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.080329 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92999ac-9100-4f7e-9010-f9f720d5e4a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh\" (UID: \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:07.101873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.080521 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnvs\" (UniqueName: \"kubernetes.io/projected/e92999ac-9100-4f7e-9010-f9f720d5e4a7-kube-api-access-clnvs\") pod \"payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh\" (UID: \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:07.101873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.080656 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f5dc-2049-48f7-aef2-55d5e5956f5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg\" (UID: \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:07.101873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.080695 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn79b\" (UniqueName: \"kubernetes.io/projected/1530f5dc-2049-48f7-aef2-55d5e5956f5f-kube-api-access-bn79b\") pod \"payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg\" (UID: \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:07.101873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.082588 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92999ac-9100-4f7e-9010-f9f720d5e4a7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh\" (UID: \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:07.161170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.161118 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:07.187207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.187172 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f5dc-2049-48f7-aef2-55d5e5956f5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg\" (UID: \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:07.265980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.265944 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn79b\" (UniqueName: \"kubernetes.io/projected/1530f5dc-2049-48f7-aef2-55d5e5956f5f-kube-api-access-bn79b\") pod \"payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg\" (UID: \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:07.269417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.266236 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:07.269678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.269646 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:07.317662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.311737 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:07.414472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.408855 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:07.481818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.466506 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:43:07.481818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.467118 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:43:07.481818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.467168 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:07.614121 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.614085 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:07.614680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.614648 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:07.802770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.802575 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:07.802990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.802812 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:07.987551 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:07.987516 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnvs\" (UniqueName: \"kubernetes.io/projected/e92999ac-9100-4f7e-9010-f9f720d5e4a7-kube-api-access-clnvs\") pod \"payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh\" (UID: \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:08.065506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.007569 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:43:08.065506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.017839 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:08.065506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.044508 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": context deadline exceeded" Oct 29 22:43:08.065506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.045375 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:43:08.103448 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:08.082005 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.127s" Oct 29 22:43:08.176807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.176781 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:43:08.177313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.177294 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6"] Oct 29 22:43:08.215784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.167799 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:08.216436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.176242 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-plugins-config-volume\") pod \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\" (UID: \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\") " Oct 29 22:43:08.216824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.216805 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-kube-api-access-7jthf\" (UniqueName: \"kubernetes.io/projected/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-kube-api-access-7jthf\") pod \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\" (UID: \"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c\") " Oct 29 22:43:08.222432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.166762 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:08.266445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.167754 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:08.266445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.223251 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:08.266445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.176647 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-plugins-config-volume" pod "b5ea960c-2e98-4332-bb5a-6a6b38f9d63c" (UID: "b5ea960c-2e98-4332-bb5a-6a6b38f9d63c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:08.335496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.324552 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:08.335496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.326770 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:08.512004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.503882 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:08.609468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.574703 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:08.609468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.574768 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:08.921079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.920989 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:08.998420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.961603 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:08.998420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:08.961964 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-kube-api-access-7jthf" pod "b5ea960c-2e98-4332-bb5a-6a6b38f9d63c" (UID: "b5ea960c-2e98-4332-bb5a-6a6b38f9d63c"). InnerVolumeSpecName "kube-api-access-7jthf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:09.044532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.044502 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jthf\" (UniqueName: \"kubernetes.io/projected/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c-kube-api-access-7jthf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:09.061742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.059881 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:09.061742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.060249 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": context deadline exceeded" Oct 29 22:43:09.061742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.060664 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": context deadline exceeded" Oct 29 22:43:09.079312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.079284 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" Oct 29 22:43:09.231423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.197506 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.281110 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r" event={"ID":"b5ea960c-2e98-4332-bb5a-6a6b38f9d63c","Type":"ContainerDied","Data":"bda80c7dc6df271c85c061aa6cb6c0bcd3bc1cb08a4d38f56508e3873082a3cb"} Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.281272 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.281370 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.281428 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.281461 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.281527 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.255586 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.282810 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.256603 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:09.298024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.284586 5610 scope.go:122] "RemoveContainer" containerID="7408d96c74f6ac4649c68ce0ed3d012ae2a6c8b57314ed3abbb9d38f45156b6d" Oct 29 22:43:09.302412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.300265 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:09.334809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.334766 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:43:09.638903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.638858 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:09.793700 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.793585 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:09.794773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.754100 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:43:09.824635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.824567 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:09.824635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:09.824622 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:10.193471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.187837 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": context deadline exceeded" Oct 29 22:43:10.193471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.188198 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:43:10.217597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.217551 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:10.299459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.299419 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:43:10.299706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.299677 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:10.485029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.403045 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:10.485029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.403116 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:10.485029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.404472 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:10.485029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.406593 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:43:10.907919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.878903 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:43:10.907919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.878980 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:10.907919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.880579 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:10.907919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.880624 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:10.907919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.907825 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:10.967190 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.967102 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:10.967525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.967485 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:10.991481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:10.991433 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:11.144456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.132767 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:11.373715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.372641 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:11.373715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.372743 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:11.433416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.426738 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:11.451501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.438915 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:11.731443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.719860 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Oct 29 22:43:11.731443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.720182 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:11.938417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.916481 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:11.959599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.946017 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:11.959599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:11.946088 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:12.018430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.007801 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:12.018430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.007867 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:12.115450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.115379 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:12.470472 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:12.427596 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.796s" Oct 29 22:43:12.572420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.475575 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:43:12.572420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.530143 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:12.661430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.660873 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:43:12.661430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.661021 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:43:12.764513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.764411 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": context deadline exceeded" Oct 29 22:43:12.799418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.754328 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:12.909417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.908596 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:12.909417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.875423 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" podUID="10fcae09-de89-49fc-98f1-3bdaf39751b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:12.931822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.909874 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 29 22:43:12.931822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.909933 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 29 22:43:12.931822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.928526 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:12.931822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.928651 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:12.931822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:12.928689 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:13.121600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.105070 5610 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-vn7xs container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:43:13.121600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.105135 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vn7xs" podUID="d0c9a67d-21dc-44cf-ab1f-1ef20fba70af" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.12:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:13.236423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.231614 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:13.282480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.264749 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" podUID="e4ac344d-ac0e-4a85-9f8e-00d4e52219bf" containerName="payload-processing" containerID="cri-o://e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818" gracePeriod=25 Oct 29 22:43:13.282480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.265476 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:13.282480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.265525 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:13.532432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.530531 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:13.607773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.607633 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:13.707055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.706760 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg"] Oct 29 22:43:13.707485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.707442 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:43:13.707561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.707499 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:43:13.808110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.808074 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 29 22:43:13.809125 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.809088 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 29 22:43:13.810125 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.810098 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:13.819444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.816494 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:43:13.819444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.817212 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:43:13.819444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:13.817462 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:43:14.054445 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:14.053523 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:43:14.159872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.159841 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 29 22:43:14.160177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.160144 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 29 22:43:14.237785 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.237697 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" event={"ID":"1530f5dc-2049-48f7-aef2-55d5e5956f5f","Type":"ContainerStarted","Data":"50989d468b8b85dcf496447b30f8cf184ea1f9a7b707bed0d32c28f0e9b91346"} Oct 29 22:43:14.238020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.237997 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" event={"ID":"e92999ac-9100-4f7e-9010-f9f720d5e4a7","Type":"ContainerStarted","Data":"e95e57d806ba72805f0b52c800defdbf2cddab4973150503b915edba5dfaa432"} Oct 29 22:43:14.238204 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.238190 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:14.254414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.238562 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:43:14.254414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.238603 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r"] Oct 29 22:43:14.254414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.238631 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-433712-69d496844c-sfn6r"] Oct 29 22:43:14.254414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.238676 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:43:14.254414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.238698 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh"] Oct 29 22:43:14.254414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.241956 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:14.254414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.243353 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:43:14.254414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.245621 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:43:14.291059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.291030 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:43:14.415635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.414698 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:43:14.426369 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:14.426154 5610 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ac344d_ac0e_4a85_9f8e_00d4e52219bf.slice/crio-conmon-e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ac344d_ac0e_4a85_9f8e_00d4e52219bf.slice/crio-e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:43:14.729266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.728463 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:14.729783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.729441 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:14.731363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.731245 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:43:14.792207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.790237 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:43:14.843282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.843200 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-plugins-config-volume\") pod \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\" (UID: \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\") " Oct 29 22:43:14.843537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.843482 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-kube-api-access-4xrqq\" (UniqueName: \"kubernetes.io/projected/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-kube-api-access-4xrqq\") pod \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\" (UID: \"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf\") " Oct 29 22:43:14.843669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.843646 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-plugins-config-volume" pod "e4ac344d-ac0e-4a85-9f8e-00d4e52219bf" (UID: "e4ac344d-ac0e-4a85-9f8e-00d4e52219bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:14.844584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.844560 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:14.846663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.846642 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-kube-api-access-4xrqq" pod "e4ac344d-ac0e-4a85-9f8e-00d4e52219bf" (UID: "e4ac344d-ac0e-4a85-9f8e-00d4e52219bf"). InnerVolumeSpecName "kube-api-access-4xrqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:14.897476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.897436 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d"] Oct 29 22:43:14.898880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.898848 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b5ea960c-2e98-4332-bb5a-6a6b38f9d63c" containerName="payload-pre-processing" Oct 29 22:43:14.898880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.898879 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ea960c-2e98-4332-bb5a-6a6b38f9d63c" containerName="payload-pre-processing" Oct 29 22:43:14.899102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.898896 5610 cpu_manager.go:414] "containerMap: removing container" podUID="e4ac344d-ac0e-4a85-9f8e-00d4e52219bf" containerName="payload-processing" Oct 29 22:43:14.899102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.898908 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ac344d-ac0e-4a85-9f8e-00d4e52219bf" containerName="payload-processing" Oct 29 22:43:14.899216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.899191 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ea960c-2e98-4332-bb5a-6a6b38f9d63c" containerName="payload-pre-processing" Oct 29 22:43:14.899216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.899204 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ac344d-ac0e-4a85-9f8e-00d4e52219bf" containerName="payload-processing" Oct 29 22:43:14.904728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.904706 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.925977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.925943 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d478b36e-openshift-default-dockercfg-6hzxg\"" Oct 29 22:43:14.930246 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.929977 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d"] Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.958557 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.958608 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.958661 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv6nf\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-kube-api-access-gv6nf\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.958757 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.959129 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.959209 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.959255 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.959318 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eebe77-a611-4c24-b5a6-a7437eefb088-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.964418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.959373 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:14.972462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:14.966207 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xrqq\" (UniqueName: \"kubernetes.io/projected/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf-kube-api-access-4xrqq\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:15.029889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.029850 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" event={"ID":"e92999ac-9100-4f7e-9010-f9f720d5e4a7","Type":"ContainerStarted","Data":"e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f"} Oct 29 22:43:15.030051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.029948 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:15.031239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.031213 5610 generic.go:356] "Generic (PLEG): container finished" podID="e4ac344d-ac0e-4a85-9f8e-00d4e52219bf" containerID="e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818" exitCode=0 Oct 29 22:43:15.031360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.031249 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" Oct 29 22:43:15.031360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.031299 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" event={"ID":"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf","Type":"ContainerDied","Data":"e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818"} Oct 29 22:43:15.031360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.031347 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6" event={"ID":"e4ac344d-ac0e-4a85-9f8e-00d4e52219bf","Type":"ContainerDied","Data":"cf2da527c5885205f129e651c20989d67affc26de6ef1753a5589413e27a5acb"} Oct 29 22:43:15.031528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.031373 5610 scope.go:122] "RemoveContainer" containerID="e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818" Oct 29 22:43:15.033366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.033338 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" event={"ID":"1530f5dc-2049-48f7-aef2-55d5e5956f5f","Type":"ContainerStarted","Data":"fd75b80eb24ff752a93de1a80bb8542190c904c2c3bb45828c34490d66995cfa"} Oct 29 22:43:15.033668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.033642 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:15.050045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.050026 5610 scope.go:122] "RemoveContainer" containerID="e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818" Oct 29 22:43:15.050375 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:15.050351 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818\": container with ID starting with e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818 not found: ID does not exist" containerID="e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818" Oct 29 22:43:15.050463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.050382 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818"} err="failed to get container status \"e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818\": rpc error: code = NotFound desc = could not find container \"e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818\": container with ID starting with e7192bc142142c1aa252dfb92d1eb1f43d8461c287be72229a552796fd59e818 not found: ID does not exist" Oct 29 22:43:15.067746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.067703 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.067922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.067777 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.067922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.067839 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eebe77-a611-4c24-b5a6-a7437eefb088-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.067933 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.068074 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.068100 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068209 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.068152 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv6nf\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-kube-api-access-gv6nf\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.068228 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-credential-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.068617 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.068687 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-socket\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.068711 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-data\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.068993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.068906 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.069336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.069298 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-certs\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.069821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.069795 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eebe77-a611-4c24-b5a6-a7437eefb088-istiod-ca-cert\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.071459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.071428 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-envoy\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.071903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.071878 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-podinfo\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.077562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.077511 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" podStartSLOduration=11.077493072 podStartE2EDuration="11.077493072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:15.073861366 +0000 UTC m=+1657.555178860" watchObservedRunningTime="2026-10-29 22:43:15.077493072 +0000 UTC m=+1657.558810565" Oct 29 22:43:15.090351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.087743 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv6nf\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-kube-api-access-gv6nf\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.107668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.107597 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-token\") pod \"e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.114149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.114081 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" podStartSLOduration=11.11406654 podStartE2EDuration="11.11406654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:15.111098868 +0000 UTC m=+1657.592416362" watchObservedRunningTime="2026-10-29 22:43:15.11406654 +0000 UTC m=+1657.595384030" Oct 29 22:43:15.147828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.147797 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6"] Oct 29 22:43:15.152147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.152116 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c06e05-fcc4d87f8-mdhg6"] Oct 29 22:43:15.163411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.163374 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:15.163910 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.163893 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:15.164634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.164610 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:43:15.237930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.237899 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:15.426481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.413520 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:15.726982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.726939 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ea960c-2e98-4332-bb5a-6a6b38f9d63c" path="/var/lib/kubelet/pods/b5ea960c-2e98-4332-bb5a-6a6b38f9d63c/volumes" Oct 29 22:43:15.727948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:15.727797 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ac344d-ac0e-4a85-9f8e-00d4e52219bf" path="/var/lib/kubelet/pods/e4ac344d-ac0e-4a85-9f8e-00d4e52219bf/volumes" Oct 29 22:43:16.071261 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:43:16.071233 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64eebe77_a611_4c24_b5a6_a7437eefb088.slice/crio-bc6780f42c6f14b5654b1a52bb9709e5d45658f2526c5a6cbc5f511d31caf7cc WatchSource:0}: Error finding container bc6780f42c6f14b5654b1a52bb9709e5d45658f2526c5a6cbc5f511d31caf7cc: Status 404 returned error can't find the container with id bc6780f42c6f14b5654b1a52bb9709e5d45658f2526c5a6cbc5f511d31caf7cc Oct 29 22:43:16.072065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:16.072040 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d"] Oct 29 22:43:16.075187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:16.075150 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:16.075295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:16.075234 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:16.075295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:16.075279 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:16.747418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:16.739033 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:17.135301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:17.135204 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" event={"ID":"64eebe77-a611-4c24-b5a6-a7437eefb088","Type":"ContainerStarted","Data":"e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab"} Oct 29 22:43:17.135978 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:17.135948 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" event={"ID":"64eebe77-a611-4c24-b5a6-a7437eefb088","Type":"ContainerStarted","Data":"bc6780f42c6f14b5654b1a52bb9709e5d45658f2526c5a6cbc5f511d31caf7cc"} Oct 29 22:43:17.220531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:17.220449 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podStartSLOduration=8.220431721 podStartE2EDuration="8.220431721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:17.216629716 +0000 UTC m=+1659.697947223" watchObservedRunningTime="2026-10-29 22:43:17.220431721 +0000 UTC m=+1659.701749214" Oct 29 22:43:17.238779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:17.238747 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:17.240505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:17.240481 5610 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:43:17.240641 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:17.240536 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:43:17.721231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:17.717909 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:18.238798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:18.238765 5610 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:43:18.239198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:18.238816 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:43:19.238693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:19.238659 5610 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:43:19.238877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:19.238721 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:43:19.867414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:19.865872 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:20.239124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:20.239084 5610 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:43:20.239517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:20.239142 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:43:21.239229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:21.239195 5610 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:43:21.239617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:21.239243 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:43:21.449511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:21.449443 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:22.239136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:22.239102 5610 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 29 22:43:22.239295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:22.239158 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 29 22:43:23.243562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:23.243528 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:23.244102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:23.244083 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:23.244944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:23.244923 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:43:23.339040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:23.338989 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:25.007287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:25.001677 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:26.047521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:26.047492 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:43:26.048134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:26.047558 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:43:26.714049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:26.714000 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:28.267942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:28.267901 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:29.241515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:29.241477 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w"] Oct 29 22:43:29.447130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:29.447102 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:29.605095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:29.605007 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:29.942081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:29.941980 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:32.189605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:32.189571 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:34.229940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.229626 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:34.244875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.244828 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" podUID="459197a3-936b-4f21-9872-521530fc1247" containerName="payload-processing" containerID="cri-o://b77d0a8891d0b13d5449bdc3aac6267f19444f70ad3b6bf63fc93690f9debd06" gracePeriod=25 Oct 29 22:43:34.450778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.450748 5610 generic.go:356] "Generic (PLEG): container finished" podID="459197a3-936b-4f21-9872-521530fc1247" containerID="b77d0a8891d0b13d5449bdc3aac6267f19444f70ad3b6bf63fc93690f9debd06" exitCode=0 Oct 29 22:43:34.451041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.450915 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" event={"ID":"459197a3-936b-4f21-9872-521530fc1247","Type":"ContainerDied","Data":"b77d0a8891d0b13d5449bdc3aac6267f19444f70ad3b6bf63fc93690f9debd06"} Oct 29 22:43:34.667418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.655453 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:43:34.730438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.728137 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459197a3-936b-4f21-9872-521530fc1247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459197a3-936b-4f21-9872-521530fc1247-plugins-config-volume\") pod \"459197a3-936b-4f21-9872-521530fc1247\" (UID: \"459197a3-936b-4f21-9872-521530fc1247\") " Oct 29 22:43:34.730438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.728273 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459197a3-936b-4f21-9872-521530fc1247-kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/459197a3-936b-4f21-9872-521530fc1247-kube-api-access-sbs8v\") pod \"459197a3-936b-4f21-9872-521530fc1247\" (UID: \"459197a3-936b-4f21-9872-521530fc1247\") " Oct 29 22:43:34.730438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.729791 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459197a3-936b-4f21-9872-521530fc1247-plugins-config-volume" pod "459197a3-936b-4f21-9872-521530fc1247" (UID: "459197a3-936b-4f21-9872-521530fc1247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:34.731868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.731845 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459197a3-936b-4f21-9872-521530fc1247-kube-api-access-sbs8v" pod "459197a3-936b-4f21-9872-521530fc1247" (UID: "459197a3-936b-4f21-9872-521530fc1247"). InnerVolumeSpecName "kube-api-access-sbs8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:34.830532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.830448 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbs8v\" (UniqueName: \"kubernetes.io/projected/459197a3-936b-4f21-9872-521530fc1247-kube-api-access-sbs8v\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:34.830532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:34.830477 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459197a3-936b-4f21-9872-521530fc1247-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:35.487947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:35.487910 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" event={"ID":"459197a3-936b-4f21-9872-521530fc1247","Type":"ContainerDied","Data":"5f7960f0bf2f0f0b71813b9b2fbc20186b3bcb9709be38f1dd1bb5facc9277bb"} Oct 29 22:43:35.488781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:35.488755 5610 scope.go:122] "RemoveContainer" containerID="b77d0a8891d0b13d5449bdc3aac6267f19444f70ad3b6bf63fc93690f9debd06" Oct 29 22:43:35.489050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:35.488546 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w" Oct 29 22:43:35.553744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:35.553706 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w"] Oct 29 22:43:35.600821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:35.600785 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1ae775-5b67fb4bfd-2829w"] Oct 29 22:43:35.645376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:35.645344 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459197a3-936b-4f21-9872-521530fc1247" path="/var/lib/kubelet/pods/459197a3-936b-4f21-9872-521530fc1247/volumes" Oct 29 22:43:35.967435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:35.962662 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:37.519841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:37.510866 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:38.196199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:38.196157 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:39.391099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:39.379958 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:40.029889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.029852 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9020-3e647b"] Oct 29 22:43:40.030847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.030829 5610 cpu_manager.go:414] "containerMap: removing container" podUID="459197a3-936b-4f21-9872-521530fc1247" containerName="payload-processing" Oct 29 22:43:40.030908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.030849 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459197a3-936b-4f21-9872-521530fc1247" containerName="payload-processing" Oct 29 22:43:40.031018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.031007 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="459197a3-936b-4f21-9872-521530fc1247" containerName="payload-processing" Oct 29 22:43:40.036300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.036268 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:40.054118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.054094 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9020-3e647b"] Oct 29 22:43:40.061021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.060996 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c5k5w\"" Oct 29 22:43:40.141524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.141491 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6x4v\" (UniqueName: \"kubernetes.io/projected/a3b87193-4fd8-4dff-97ff-28af3c1277a0-kube-api-access-h6x4v\") pod \"test-curl-9020-3e647b\" (UID: \"a3b87193-4fd8-4dff-97ff-28af3c1277a0\") " pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:40.243411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.243362 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6x4v\" (UniqueName: \"kubernetes.io/projected/a3b87193-4fd8-4dff-97ff-28af3c1277a0-kube-api-access-h6x4v\") pod \"test-curl-9020-3e647b\" (UID: \"a3b87193-4fd8-4dff-97ff-28af3c1277a0\") " pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:40.262881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.262847 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6x4v\" (UniqueName: \"kubernetes.io/projected/a3b87193-4fd8-4dff-97ff-28af3c1277a0-kube-api-access-h6x4v\") pod \"test-curl-9020-3e647b\" (UID: \"a3b87193-4fd8-4dff-97ff-28af3c1277a0\") " pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:40.353469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.353439 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:40.637931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.637841 5610 generic.go:356] "Generic (PLEG): container finished" podID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerID="157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e" exitCode=0 Oct 29 22:43:40.637931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.637908 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" event={"ID":"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36","Type":"ContainerDied","Data":"157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e"} Oct 29 22:43:40.638507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.638486 5610 scope.go:122] "RemoveContainer" containerID="157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e" Oct 29 22:43:40.640546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.640511 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:40.640690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.640602 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:40.640690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.640652 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:40.672827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.672803 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9020-3e647b"] Oct 29 22:43:40.917524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:40.917375 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:41.667416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:41.663524 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" event={"ID":"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36","Type":"ContainerStarted","Data":"b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2"} Oct 29 22:43:41.667416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:41.666617 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9020-3e647b" event={"ID":"a3b87193-4fd8-4dff-97ff-28af3c1277a0","Type":"ContainerStarted","Data":"743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5"} Oct 29 22:43:41.667416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:41.666658 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9020-3e647b" event={"ID":"a3b87193-4fd8-4dff-97ff-28af3c1277a0","Type":"ContainerStarted","Data":"4a4744dc8ca3aa16b09d2ee20ec0d20dc48c90c2876f7a29b322267cfb9141b3"} Oct 29 22:43:41.935467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:41.934755 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:43:41.942868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:41.942836 5610 patch_prober.go:29] interesting pod/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:43:41.943054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:41.942886 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 29 22:43:42.687419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:42.686498 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9020-3e647b" podStartSLOduration=2.686476628 podStartE2EDuration="2.686476628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:41.791572483 +0000 UTC m=+1684.272890057" watchObservedRunningTime="2026-10-29 22:43:42.686476628 +0000 UTC m=+1685.167794123" Oct 29 22:43:42.700289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:42.699824 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9020-3e647b"] Oct 29 22:43:42.712791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:42.708268 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9020-3e647b"] Oct 29 22:43:42.841547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:42.825005 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:42.951469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:42.950953 5610 patch_prober.go:29] interesting pod/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:43:42.951469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:42.951028 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 29 22:43:43.715517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:43.715480 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9020-3e647b" podUID="a3b87193-4fd8-4dff-97ff-28af3c1277a0" containerName="test-curl-9020-3e647b" containerID="cri-o://743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5" gracePeriod=2 Oct 29 22:43:43.729291 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:43.729256 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:43:43.932715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:43.932681 5610 patch_prober.go:29] interesting pod/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:43:43.932886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:43.932733 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 29 22:43:44.775415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:44.767837 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:44.930171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:44.930120 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:43:44.932449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:44.932412 5610 patch_prober.go:29] interesting pod/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:43:44.932642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:44.932509 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 29 22:43:45.932764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:45.932729 5610 patch_prober.go:29] interesting pod/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:43:45.933222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:45.932789 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 29 22:43:46.031704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.031678 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:46.148538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.148504 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b87193-4fd8-4dff-97ff-28af3c1277a0-kube-api-access-h6x4v\" (UniqueName: \"kubernetes.io/projected/a3b87193-4fd8-4dff-97ff-28af3c1277a0-kube-api-access-h6x4v\") pod \"a3b87193-4fd8-4dff-97ff-28af3c1277a0\" (UID: \"a3b87193-4fd8-4dff-97ff-28af3c1277a0\") " Oct 29 22:43:46.150702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.150672 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b87193-4fd8-4dff-97ff-28af3c1277a0-kube-api-access-h6x4v" pod "a3b87193-4fd8-4dff-97ff-28af3c1277a0" (UID: "a3b87193-4fd8-4dff-97ff-28af3c1277a0"). InnerVolumeSpecName "kube-api-access-h6x4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:46.251803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.251726 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6x4v\" (UniqueName: \"kubernetes.io/projected/a3b87193-4fd8-4dff-97ff-28af3c1277a0-kube-api-access-h6x4v\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:46.688464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.681039 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:46.918416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.910485 5610 generic.go:356] "Generic (PLEG): container finished" podID="a3b87193-4fd8-4dff-97ff-28af3c1277a0" containerID="743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5" exitCode=137 Oct 29 22:43:46.918416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.910530 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:46.918416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.910561 5610 scope.go:122] "RemoveContainer" containerID="743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5" Oct 29 22:43:46.926022 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.926005 5610 scope.go:122] "RemoveContainer" containerID="743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5" Oct 29 22:43:46.926278 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:46.926262 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5\": container with ID starting with 743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5 not found: ID does not exist" containerID="743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5" Oct 29 22:43:46.926329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.926285 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5"} err="failed to get container status \"743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5\": rpc error: code = NotFound desc = could not find container \"743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5\": container with ID starting with 743c430592ec39a18bc24e8cfd8881b286a84e64fc17e7959448ae2be6550fc5 not found: ID does not exist" Oct 29 22:43:46.932730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.932691 5610 patch_prober.go:29] interesting pod/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 29 22:43:46.932897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:46.932874 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 29 22:43:46.940015 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:46.939888 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9020-3e647b\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="a3b87193-4fd8-4dff-97ff-28af3c1277a0" pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:47.640331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:47.640293 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b87193-4fd8-4dff-97ff-28af3c1277a0" path="/var/lib/kubelet/pods/a3b87193-4fd8-4dff-97ff-28af3c1277a0/volumes" Oct 29 22:43:47.649552 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:47.649518 5610 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9020-3e647b\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="a3b87193-4fd8-4dff-97ff-28af3c1277a0" pod="opendatahub/test-curl-9020-3e647b" Oct 29 22:43:47.657589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:47.657563 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:47.793733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:47.793698 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:47.936995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:47.936920 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:43:47.938167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:47.938142 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:43:48.298029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:48.297983 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:48.436011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:48.435959 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:48.598003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:48.597908 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:49.075207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:49.075166 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:49.230419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:49.223958 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:50.211803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:50.211767 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:50.655692 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:50.655601 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn"] Oct 29 22:43:50.655878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:50.655850 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" containerID="cri-o://228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea" gracePeriod=30 Oct 29 22:43:52.016566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:52.016532 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:53.417883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:53.417847 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4"] Oct 29 22:43:53.461760 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:53.461716 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:53.610540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:53.610507 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:53.774258 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:53.774215 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:55.181028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.180989 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:55.540467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.540435 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:43:55.540656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.540490 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:43:55.980690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.980662 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:43:55.994903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.994871 5610 generic.go:356] "Generic (PLEG): container finished" podID="6a7a8eca-7072-46d9-964f-145ef0530128" containerID="228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea" exitCode=0 Oct 29 22:43:55.995055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.994917 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" Oct 29 22:43:55.995055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.994925 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" event={"ID":"6a7a8eca-7072-46d9-964f-145ef0530128","Type":"ContainerDied","Data":"228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea"} Oct 29 22:43:55.995055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.994967 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn" event={"ID":"6a7a8eca-7072-46d9-964f-145ef0530128","Type":"ContainerDied","Data":"ff4d6db072e6803ba1dcfb007dbfb8614d0333d090a6ee4b94208648c0153eec"} Oct 29 22:43:55.995055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:55.994988 5610 scope.go:122] "RemoveContainer" containerID="228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea" Oct 29 22:43:56.006980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.006961 5610 scope.go:122] "RemoveContainer" containerID="228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea" Oct 29 22:43:56.007276 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:56.007254 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea\": container with ID starting with 228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea not found: ID does not exist" containerID="228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea" Oct 29 22:43:56.007346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.007288 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea"} err="failed to get container status \"228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea\": rpc error: code = NotFound desc = could not find container \"228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea\": container with ID starting with 228223d87cc083d448ca91aba0dc5afe227ec4c7caa6bfd6efead917c873b2ea not found: ID does not exist" Oct 29 22:43:56.061448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.061411 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b"] Oct 29 22:43:56.062888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.062861 5610 cpu_manager.go:414] "containerMap: removing container" podUID="a3b87193-4fd8-4dff-97ff-28af3c1277a0" containerName="test-curl-9020-3e647b" Oct 29 22:43:56.063063 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.063038 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b87193-4fd8-4dff-97ff-28af3c1277a0" containerName="test-curl-9020-3e647b" Oct 29 22:43:56.063215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.063143 5610 cpu_manager.go:414] "containerMap: removing container" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" Oct 29 22:43:56.063215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.063151 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" Oct 29 22:43:56.063358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.063318 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" containerName="istio-proxy" Oct 29 22:43:56.063358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.063336 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b87193-4fd8-4dff-97ff-28af3c1277a0" containerName="test-curl-9020-3e647b" Oct 29 22:43:56.067996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.067973 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.076439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.076388 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b"] Oct 29 22:43:56.083477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.083452 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-379a6d-openshift-default-dockercfg-zn4lk\"" Oct 29 22:43:56.102440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.102412 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-envoy\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.102598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.102523 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a7a8eca-7072-46d9-964f-145ef0530128-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7a8eca-7072-46d9-964f-145ef0530128-istiod-ca-cert\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.102665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.102606 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-istio-token\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-istio-token\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.103018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.102724 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-kube-api-access-fdszl\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-kube-api-access-fdszl\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.103018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.102777 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-certs\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.103018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.102831 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-data\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.103018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.102969 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-socket\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.103018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.102988 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7a8eca-7072-46d9-964f-145ef0530128-istiod-ca-cert" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:56.103376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.103060 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-credential-socket\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.103376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.103144 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a7a8eca-7072-46d9-964f-145ef0530128-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7a8eca-7072-46d9-964f-145ef0530128-istio-podinfo\") pod \"6a7a8eca-7072-46d9-964f-145ef0530128\" (UID: \"6a7a8eca-7072-46d9-964f-145ef0530128\") " Oct 29 22:43:56.103526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.103343 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-data" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.103526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.103504 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-envoy\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.103625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.103548 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-credential-socket" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.103819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.103792 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4578473-cc01-462c-920b-c9e76538ca5e-istio-podinfo\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.103926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.103902 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-credential-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.103987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.103945 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-certs\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.104112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104034 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7dj\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-kube-api-access-qj7dj\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.104186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104127 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.104186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104163 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4578473-cc01-462c-920b-c9e76538ca5e-istiod-ca-cert\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.104287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104237 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-data\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.104348 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104287 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-istio-token\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.104450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104432 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.104525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104459 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.104525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104474 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a7a8eca-7072-46d9-964f-145ef0530128-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.104691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104669 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-socket" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.105735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.104983 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-certs" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.105735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.105709 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-kube-api-access-fdszl" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "kube-api-access-fdszl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:56.106005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.105975 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-envoy" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:43:56.106463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.106441 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a7a8eca-7072-46d9-964f-145ef0530128-istio-podinfo" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:43:56.107353 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.107335 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-istio-token" pod "6a7a8eca-7072-46d9-964f-145ef0530128" (UID: "6a7a8eca-7072-46d9-964f-145ef0530128"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:56.205709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.205675 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4578473-cc01-462c-920b-c9e76538ca5e-istio-podinfo\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.205765 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-credential-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.205796 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-certs\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.205854 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7dj\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-kube-api-access-qj7dj\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.205916 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.205943 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4578473-cc01-462c-920b-c9e76538ca5e-istiod-ca-cert\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206008 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-data\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206055 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-istio-token\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206112 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-envoy\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206233 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a7a8eca-7072-46d9-964f-145ef0530128-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206254 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206270 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206282 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdszl\" (UniqueName: \"kubernetes.io/projected/6a7a8eca-7072-46d9-964f-145ef0530128-kube-api-access-fdszl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206287 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-credential-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206307 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-certs\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206297 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.206339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206342 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a7a8eca-7072-46d9-964f-145ef0530128-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:56.207446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206597 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-data\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.207446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.206770 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-socket\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.207446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.207104 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4578473-cc01-462c-920b-c9e76538ca5e-istiod-ca-cert\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.208326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.208296 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-envoy\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.208665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.208643 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4578473-cc01-462c-920b-c9e76538ca5e-istio-podinfo\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.225069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.225045 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7dj\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-kube-api-access-qj7dj\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.225181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.225043 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-istio-token\") pod \"e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.343084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.343048 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn"] Oct 29 22:43:56.347955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.347926 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-xndxn"] Oct 29 22:43:56.392230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.392149 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:56.718350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.718168 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b"] Oct 29 22:43:56.718638 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:43:56.718615 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4578473_cc01_462c_920b_c9e76538ca5e.slice/crio-30619d547ec2ec47390ca33d06b7206b8d27624888bfb73b9d6cd06dc4414575 WatchSource:0}: Error finding container 30619d547ec2ec47390ca33d06b7206b8d27624888bfb73b9d6cd06dc4414575: Status 404 returned error can't find the container with id 30619d547ec2ec47390ca33d06b7206b8d27624888bfb73b9d6cd06dc4414575 Oct 29 22:43:56.722362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.722320 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:56.722498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.722427 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:56.722498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.722473 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:56.879539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:56.879501 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:57.073493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:57.067107 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" event={"ID":"c4578473-cc01-462c-920b-c9e76538ca5e","Type":"ContainerStarted","Data":"96c460cc54a909d9fec5f1973122e2a649c28de502549c41c7e4fc1f1bea015e"} Oct 29 22:43:57.073493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:57.067193 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" event={"ID":"c4578473-cc01-462c-920b-c9e76538ca5e","Type":"ContainerStarted","Data":"30619d547ec2ec47390ca33d06b7206b8d27624888bfb73b9d6cd06dc4414575"} Oct 29 22:43:57.131414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:57.130509 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podStartSLOduration=9.130487206 podStartE2EDuration="9.130487206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:43:57.119960971 +0000 UTC m=+1699.601278466" watchObservedRunningTime="2026-10-29 22:43:57.130487206 +0000 UTC m=+1699.611804700" Oct 29 22:43:57.415652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:57.406312 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:57.416359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:57.416329 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:43:57.416595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:57.416563 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 29 22:43:57.664006 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:57.663884 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a7a8eca-7072-46d9-964f-145ef0530128" path="/var/lib/kubelet/pods/6a7a8eca-7072-46d9-964f-145ef0530128/volumes" Oct 29 22:43:58.397986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.397952 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:58.419027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.418972 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" podUID="9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6" containerName="payload-processing" containerID="cri-o://5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369" gracePeriod=25 Oct 29 22:43:58.738020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.736324 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:58.835009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.834968 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:43:58.892705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.892652 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p"] Oct 29 22:43:58.894028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.894004 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6" containerName="payload-processing" Oct 29 22:43:58.894093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.894033 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6" containerName="payload-processing" Oct 29 22:43:58.894311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.894297 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6" containerName="payload-processing" Oct 29 22:43:58.897844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.897819 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:58.914760 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.914506 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-63db1a-openshift-default-dockercfg-x2525\"" Oct 29 22:43:58.924530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.924505 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p"] Oct 29 22:43:58.938417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.938375 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-plugins-config-volume\") pod \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\" (UID: \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\") " Oct 29 22:43:58.938893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.938850 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-kube-api-access-2xltp\" (UniqueName: \"kubernetes.io/projected/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-kube-api-access-2xltp\") pod \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\" (UID: \"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6\") " Oct 29 22:43:58.939287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.939012 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-plugins-config-volume" pod "9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6" (UID: "9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:43:58.939942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.939919 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:58.942090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:58.942051 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-kube-api-access-2xltp" pod "9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6" (UID: "9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6"). InnerVolumeSpecName "kube-api-access-2xltp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:43:59.041472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.041326 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147828d4-7a3e-4f50-964f-7132b042aec2-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.041472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.041403 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147828d4-7a3e-4f50-964f-7132b042aec2-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.041472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.041469 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.041733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.041508 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.041733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.041565 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.041733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.041655 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.041733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.041719 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.041942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.041911 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8z9g\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-kube-api-access-z8z9g\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.042032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.042008 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.042168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.042155 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xltp\" (UniqueName: \"kubernetes.io/projected/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6-kube-api-access-2xltp\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:43:59.107743 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.107695 5610 generic.go:356] "Generic (PLEG): container finished" podID="9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6" containerID="5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369" exitCode=0 Oct 29 22:43:59.107946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.107753 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" Oct 29 22:43:59.107946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.107799 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" event={"ID":"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6","Type":"ContainerDied","Data":"5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369"} Oct 29 22:43:59.107946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.107880 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4" event={"ID":"9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6","Type":"ContainerDied","Data":"2f018b8ce7664247f4fd3b00998cb63a3d679e1ddf064128c428c74b9d627450"} Oct 29 22:43:59.107946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.107910 5610 scope.go:122] "RemoveContainer" containerID="5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369" Oct 29 22:43:59.108833 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.108798 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:59.110963 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.110936 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:43:59.121755 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.121738 5610 scope.go:122] "RemoveContainer" containerID="5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369" Oct 29 22:43:59.122041 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:43:59.122019 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369\": container with ID starting with 5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369 not found: ID does not exist" containerID="5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369" Oct 29 22:43:59.122139 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.122046 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369"} err="failed to get container status \"5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369\": rpc error: code = NotFound desc = could not find container \"5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369\": container with ID starting with 5e46a585111f4e230909a909f73485dded0b044974921c01d8c3f06e42c12369 not found: ID does not exist" Oct 29 22:43:59.143709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.143687 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.143826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.143813 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8z9g\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-kube-api-access-z8z9g\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.143871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.143862 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.143962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.143943 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147828d4-7a3e-4f50-964f-7132b042aec2-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.143976 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147828d4-7a3e-4f50-964f-7132b042aec2-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.143994 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.144019 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.144052 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.144057 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.144118 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.144407 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.144491 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.144511 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.144698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.144682 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147828d4-7a3e-4f50-964f-7132b042aec2-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.146330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.146311 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.146409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.146329 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147828d4-7a3e-4f50-964f-7132b042aec2-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.171676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.171642 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8z9g\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-kube-api-access-z8z9g\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.177953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.177879 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.210374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.210337 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4"] Oct 29 22:43:59.214276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.214024 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:43:59.217536 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.217366 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d478b36e-65b9f9c44b-wrgq4"] Oct 29 22:43:59.584625 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:43:59.584521 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147828d4_7a3e_4f50_964f_7132b042aec2.slice/crio-b763bdaa56b746d3ad2ab6d856fdadaaffdd27c693b79e7526bc89c52ef78609 WatchSource:0}: Error finding container b763bdaa56b746d3ad2ab6d856fdadaaffdd27c693b79e7526bc89c52ef78609: Status 404 returned error can't find the container with id b763bdaa56b746d3ad2ab6d856fdadaaffdd27c693b79e7526bc89c52ef78609 Oct 29 22:43:59.586640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.586608 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p"] Oct 29 22:43:59.587514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.587479 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:59.587639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.587548 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:59.587639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.587579 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:43:59.636267 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.636226 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:43:59.640643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.640616 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6" path="/var/lib/kubelet/pods/9d34b5d6-eb5c-4dc9-b6f8-8434d71331e6/volumes" Oct 29 22:43:59.822185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:43:59.822150 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:00.122815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:00.122769 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" event={"ID":"147828d4-7a3e-4f50-964f-7132b042aec2","Type":"ContainerStarted","Data":"0fe7244e3e5912626afd9048cf31cc54ca188269988f5b3842b9146a477bab36"} Oct 29 22:44:00.122975 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:00.122878 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" event={"ID":"147828d4-7a3e-4f50-964f-7132b042aec2","Type":"ContainerStarted","Data":"b763bdaa56b746d3ad2ab6d856fdadaaffdd27c693b79e7526bc89c52ef78609"} Oct 29 22:44:00.198537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:00.198443 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podStartSLOduration=42.198422937 podStartE2EDuration="42.198422937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:00.188740211 +0000 UTC m=+1702.670057705" watchObservedRunningTime="2026-10-29 22:44:00.198422937 +0000 UTC m=+1702.679740434" Oct 29 22:44:00.215179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:00.215141 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:44:00.216976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:00.216945 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:44:00.217130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:00.217003 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:44:00.670661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:00.670302 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:01.215084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:01.215060 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:44:01.215516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:01.215476 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:44:01.893081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:01.893047 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:02.082418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:02.076323 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:02.217416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:02.215778 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:44:02.217416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:02.215840 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:44:02.982413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:02.982144 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:03.215112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:03.215073 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:44:03.215307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:03.215142 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:44:04.215660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:04.215627 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:44:04.216086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:04.215685 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:44:04.718979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:04.718918 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:05.215279 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:05.215241 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 29 22:44:05.215501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:05.215289 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 29 22:44:06.351285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:06.351236 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:44:06.353270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:06.352794 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:44:06.354166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:06.353967 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:44:06.698417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:06.694545 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:08.670173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:08.670080 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:09.998792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:09.998750 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq"] Oct 29 22:44:10.000158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:09.999489 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" podUID="cb1b41de-a916-4233-8219-737eaf1377a7" containerName="maas-api" containerID="cri-o://a382c916ac220d16242d4b67e92ec98dfa4265edae67315cd602be002fddf4ee" gracePeriod=30 Oct 29 22:44:10.139724 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.139683 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:10.268100 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.268014 5610 generic.go:356] "Generic (PLEG): container finished" podID="cb1b41de-a916-4233-8219-737eaf1377a7" containerID="a382c916ac220d16242d4b67e92ec98dfa4265edae67315cd602be002fddf4ee" exitCode=0 Oct 29 22:44:10.268100 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.268071 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" event={"ID":"cb1b41de-a916-4233-8219-737eaf1377a7","Type":"ContainerDied","Data":"a382c916ac220d16242d4b67e92ec98dfa4265edae67315cd602be002fddf4ee"} Oct 29 22:44:10.351481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.351454 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:44:10.364335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.364301 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd"] Oct 29 22:44:10.364585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.364558 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" podUID="1ce6012c-043e-4bcd-b986-180c8d2f38c0" containerName="maas-api" containerID="cri-o://1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440" gracePeriod=30 Oct 29 22:44:10.407301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.407264 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1b41de-a916-4233-8219-737eaf1377a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b41de-a916-4233-8219-737eaf1377a7-openshift-service-ca\") pod \"cb1b41de-a916-4233-8219-737eaf1377a7\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " Oct 29 22:44:10.407680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.407604 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-metrics-certs\") pod \"cb1b41de-a916-4233-8219-737eaf1377a7\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " Oct 29 22:44:10.407834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.407690 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-maas-api-tls\") pod \"cb1b41de-a916-4233-8219-737eaf1377a7\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " Oct 29 22:44:10.407834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.407793 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1b41de-a916-4233-8219-737eaf1377a7-kube-api-access-7jnw9\" (UniqueName: \"kubernetes.io/projected/cb1b41de-a916-4233-8219-737eaf1377a7-kube-api-access-7jnw9\") pod \"cb1b41de-a916-4233-8219-737eaf1377a7\" (UID: \"cb1b41de-a916-4233-8219-737eaf1377a7\") " Oct 29 22:44:10.409321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.409293 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1b41de-a916-4233-8219-737eaf1377a7-openshift-service-ca" pod "cb1b41de-a916-4233-8219-737eaf1377a7" (UID: "cb1b41de-a916-4233-8219-737eaf1377a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:10.412800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.411505 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1b41de-a916-4233-8219-737eaf1377a7-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:10.420691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.420659 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1b41de-a916-4233-8219-737eaf1377a7-kube-api-access-7jnw9" pod "cb1b41de-a916-4233-8219-737eaf1377a7" (UID: "cb1b41de-a916-4233-8219-737eaf1377a7"). InnerVolumeSpecName "kube-api-access-7jnw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:10.423731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.423697 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-metrics-certs" pod "cb1b41de-a916-4233-8219-737eaf1377a7" (UID: "cb1b41de-a916-4233-8219-737eaf1377a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:10.426786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.426758 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-maas-api-tls" pod "cb1b41de-a916-4233-8219-737eaf1377a7" (UID: "cb1b41de-a916-4233-8219-737eaf1377a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:10.465698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.465536 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:10.513673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.513634 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnw9\" (UniqueName: \"kubernetes.io/projected/cb1b41de-a916-4233-8219-737eaf1377a7-kube-api-access-7jnw9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:10.513673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.513674 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:10.513921 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.513691 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1b41de-a916-4233-8219-737eaf1377a7-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:10.618177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.618052 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:10.689211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.689179 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:44:10.719077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.719041 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce6012c-043e-4bcd-b986-180c8d2f38c0-kube-api-access-hwm44\" (UniqueName: \"kubernetes.io/projected/1ce6012c-043e-4bcd-b986-180c8d2f38c0-kube-api-access-hwm44\") pod \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " Oct 29 22:44:10.719266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.719182 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce6012c-043e-4bcd-b986-180c8d2f38c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce6012c-043e-4bcd-b986-180c8d2f38c0-openshift-service-ca\") pod \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " Oct 29 22:44:10.719266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.719226 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-metrics-certs\") pod \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " Oct 29 22:44:10.719536 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.719511 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-maas-api-tls\") pod \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\" (UID: \"1ce6012c-043e-4bcd-b986-180c8d2f38c0\") " Oct 29 22:44:10.720329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.719952 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce6012c-043e-4bcd-b986-180c8d2f38c0-openshift-service-ca" pod "1ce6012c-043e-4bcd-b986-180c8d2f38c0" (UID: "1ce6012c-043e-4bcd-b986-180c8d2f38c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:10.720329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.720282 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce6012c-043e-4bcd-b986-180c8d2f38c0-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:10.721786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.721755 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-metrics-certs" pod "1ce6012c-043e-4bcd-b986-180c8d2f38c0" (UID: "1ce6012c-043e-4bcd-b986-180c8d2f38c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:10.721925 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.721903 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce6012c-043e-4bcd-b986-180c8d2f38c0-kube-api-access-hwm44" pod "1ce6012c-043e-4bcd-b986-180c8d2f38c0" (UID: "1ce6012c-043e-4bcd-b986-180c8d2f38c0"). InnerVolumeSpecName "kube-api-access-hwm44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:10.723201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.723177 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-maas-api-tls" pod "1ce6012c-043e-4bcd-b986-180c8d2f38c0" (UID: "1ce6012c-043e-4bcd-b986-180c8d2f38c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:10.730248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.730223 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2"] Oct 29 22:44:10.730528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.730478 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" podUID="030c3c93-fe58-45ce-a46e-66c36a44b801" containerName="maas-api" containerID="cri-o://a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5" gracePeriod=30 Oct 29 22:44:10.780531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.780439 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:10.821496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.821415 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:10.821496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.821447 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ce6012c-043e-4bcd-b986-180c8d2f38c0-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:10.821496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.821456 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwm44\" (UniqueName: \"kubernetes.io/projected/1ce6012c-043e-4bcd-b986-180c8d2f38c0-kube-api-access-hwm44\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:10.932986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:10.931806 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:11.083463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.082142 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:44:11.095198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.095133 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:11.134623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.134276 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-maas-api-tls\") pod \"030c3c93-fe58-45ce-a46e-66c36a44b801\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " Oct 29 22:44:11.134815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.134678 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030c3c93-fe58-45ce-a46e-66c36a44b801-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030c3c93-fe58-45ce-a46e-66c36a44b801-openshift-service-ca\") pod \"030c3c93-fe58-45ce-a46e-66c36a44b801\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " Oct 29 22:44:11.134904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.134838 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030c3c93-fe58-45ce-a46e-66c36a44b801-kube-api-access-ssvgf\" (UniqueName: \"kubernetes.io/projected/030c3c93-fe58-45ce-a46e-66c36a44b801-kube-api-access-ssvgf\") pod \"030c3c93-fe58-45ce-a46e-66c36a44b801\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " Oct 29 22:44:11.134970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.134915 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-metrics-certs\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-metrics-certs\") pod \"030c3c93-fe58-45ce-a46e-66c36a44b801\" (UID: \"030c3c93-fe58-45ce-a46e-66c36a44b801\") " Oct 29 22:44:11.145970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.140082 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030c3c93-fe58-45ce-a46e-66c36a44b801-openshift-service-ca" pod "030c3c93-fe58-45ce-a46e-66c36a44b801" (UID: "030c3c93-fe58-45ce-a46e-66c36a44b801"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:11.145970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.145011 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030c3c93-fe58-45ce-a46e-66c36a44b801-kube-api-access-ssvgf" pod "030c3c93-fe58-45ce-a46e-66c36a44b801" (UID: "030c3c93-fe58-45ce-a46e-66c36a44b801"). InnerVolumeSpecName "kube-api-access-ssvgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:11.151946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.151910 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-maas-api-tls" pod "030c3c93-fe58-45ce-a46e-66c36a44b801" (UID: "030c3c93-fe58-45ce-a46e-66c36a44b801"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:11.184420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.178535 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-metrics-certs" pod "030c3c93-fe58-45ce-a46e-66c36a44b801" (UID: "030c3c93-fe58-45ce-a46e-66c36a44b801"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:11.264418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.258157 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvgf\" (UniqueName: \"kubernetes.io/projected/030c3c93-fe58-45ce-a46e-66c36a44b801-kube-api-access-ssvgf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.264418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.258197 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.264418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.258212 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/030c3c93-fe58-45ce-a46e-66c36a44b801-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.264418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.258224 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/030c3c93-fe58-45ce-a46e-66c36a44b801-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:11.291418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.283943 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:11.393539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.386996 5610 generic.go:356] "Generic (PLEG): container finished" podID="030c3c93-fe58-45ce-a46e-66c36a44b801" containerID="a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5" exitCode=0 Oct 29 22:44:11.393539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.387211 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" event={"ID":"030c3c93-fe58-45ce-a46e-66c36a44b801","Type":"ContainerDied","Data":"a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5"} Oct 29 22:44:11.393539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.387265 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" event={"ID":"030c3c93-fe58-45ce-a46e-66c36a44b801","Type":"ContainerDied","Data":"00d980a40c87cdcaef424ece9b632974a71489ad095f523092ef6ab84f5e2ca5"} Oct 29 22:44:11.393539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.387297 5610 scope.go:122] "RemoveContainer" containerID="a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5" Oct 29 22:44:11.393539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.387572 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2" Oct 29 22:44:11.801647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.801618 5610 scope.go:122] "RemoveContainer" containerID="a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5" Oct 29 22:44:11.802305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.802278 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:11.805149 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:11.805086 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5\": container with ID starting with a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5 not found: ID does not exist" containerID="a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5" Oct 29 22:44:11.805325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.805298 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5"} err="failed to get container status \"a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5\": rpc error: code = NotFound desc = could not find container \"a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5\": container with ID starting with a6dbb0a617a928767c90a96f42ab9b5e97ef685fb3d01e3d120032d4607c9bc5 not found: ID does not exist" Oct 29 22:44:11.906234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.906143 5610 generic.go:356] "Generic (PLEG): container finished" podID="1ce6012c-043e-4bcd-b986-180c8d2f38c0" containerID="1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440" exitCode=0 Oct 29 22:44:11.906721 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.906700 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" Oct 29 22:44:11.921516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.921478 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" Oct 29 22:44:11.967337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:11.966494 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:12.027896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.027856 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" event={"ID":"1ce6012c-043e-4bcd-b986-180c8d2f38c0","Type":"ContainerDied","Data":"1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440"} Oct 29 22:44:12.035544 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.035509 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd" event={"ID":"1ce6012c-043e-4bcd-b986-180c8d2f38c0","Type":"ContainerDied","Data":"4718f60532ac03397212e71bf4d27c6d0bb49430b684302b8e785d7c3a171c17"} Oct 29 22:44:12.035793 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.035768 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq" event={"ID":"cb1b41de-a916-4233-8219-737eaf1377a7","Type":"ContainerDied","Data":"ad023bcd0c9cfddc66fd162e9d69a8714cb09602d5a42966c71895f8f77fed72"} Oct 29 22:44:12.035940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.035922 5610 scope.go:122] "RemoveContainer" containerID="1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440" Oct 29 22:44:12.045762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.045711 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2"] Oct 29 22:44:12.052439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.052417 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d478b36e-75bc765888-t8tg2"] Oct 29 22:44:12.059357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.059332 5610 scope.go:122] "RemoveContainer" containerID="1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440" Oct 29 22:44:12.059701 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:12.059673 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440\": container with ID starting with 1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440 not found: ID does not exist" containerID="1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440" Oct 29 22:44:12.059792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.059712 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440"} err="failed to get container status \"1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440\": rpc error: code = NotFound desc = could not find container \"1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440\": container with ID starting with 1476dcf7a2cc75da42c4888fce0152e971d4d6bc79903dad399a87359ff89440 not found: ID does not exist" Oct 29 22:44:12.059792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.059735 5610 scope.go:122] "RemoveContainer" containerID="a382c916ac220d16242d4b67e92ec98dfa4265edae67315cd602be002fddf4ee" Oct 29 22:44:12.074871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.074601 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq"] Oct 29 22:44:12.081329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.081297 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-433712-579589d4dc-d7nkq"] Oct 29 22:44:12.111814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.111775 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd"] Oct 29 22:44:12.117638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.117611 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06e05-7d476758fd-9jxzd"] Oct 29 22:44:12.385447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.385411 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:12.523606 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:12.523568 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:13.638516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:13.638484 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="030c3c93-fe58-45ce-a46e-66c36a44b801" path="/var/lib/kubelet/pods/030c3c93-fe58-45ce-a46e-66c36a44b801/volumes" Oct 29 22:44:13.639072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:13.639055 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce6012c-043e-4bcd-b986-180c8d2f38c0" path="/var/lib/kubelet/pods/1ce6012c-043e-4bcd-b986-180c8d2f38c0/volumes" Oct 29 22:44:13.639503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:13.639488 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1b41de-a916-4233-8219-737eaf1377a7" path="/var/lib/kubelet/pods/cb1b41de-a916-4233-8219-737eaf1377a7/volumes" Oct 29 22:44:13.712731 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:13.712696 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:44:14.199988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:14.183352 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:16.029418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:16.023119 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:17.574908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:17.574871 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:17.805419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:17.801136 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:18.597803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:18.597733 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:18.784418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:18.784339 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:19.072067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.071307 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:19.097434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.093755 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp"] Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.118773 5610 cpu_manager.go:414] "containerMap: removing container" podUID="030c3c93-fe58-45ce-a46e-66c36a44b801" containerName="maas-api" Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.118809 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030c3c93-fe58-45ce-a46e-66c36a44b801" containerName="maas-api" Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.118889 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1ce6012c-043e-4bcd-b986-180c8d2f38c0" containerName="maas-api" Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.118898 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce6012c-043e-4bcd-b986-180c8d2f38c0" containerName="maas-api" Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.118933 5610 cpu_manager.go:414] "containerMap: removing container" podUID="cb1b41de-a916-4233-8219-737eaf1377a7" containerName="maas-api" Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.118941 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1b41de-a916-4233-8219-737eaf1377a7" containerName="maas-api" Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.119168 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce6012c-043e-4bcd-b986-180c8d2f38c0" containerName="maas-api" Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.119193 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1b41de-a916-4233-8219-737eaf1377a7" containerName="maas-api" Oct 29 22:44:19.120428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.119208 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="030c3c93-fe58-45ce-a46e-66c36a44b801" containerName="maas-api" Oct 29 22:44:19.246690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.246544 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf"] Oct 29 22:44:19.249529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.246655 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.273476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.271613 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9"] Oct 29 22:44:19.282136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.282104 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m"] Oct 29 22:44:19.286329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.286258 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp"] Oct 29 22:44:19.286708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.286687 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk"] Oct 29 22:44:19.287064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.287035 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" containerID="cri-o://294f76ea4243d3f3e262919bc31fddeec97bb235c1ab7b0d1be1e440f989c341" gracePeriod=30 Oct 29 22:44:19.296088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.296058 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.296571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.296550 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.297666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.297639 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" containerID="cri-o://aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb" gracePeriod=30 Oct 29 22:44:19.298078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.298051 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" containerID="cri-o://a946d81c76a0acf46d72e19829074c17c8acd0d26f986b7b84d299d6da1ce00c" gracePeriod=30 Oct 29 22:44:19.374779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.369419 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf"] Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448501 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448565 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448598 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448645 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448671 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448754 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448787 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448813 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76lj\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-kube-api-access-k76lj\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448834 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f653887-6403-42ab-96ee-dfc750ab4be3-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448891 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448918 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a2e42-7162-459d-817a-1913f54ec483-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448940 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.448995 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449022 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449042 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449070 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449120 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a2e42-7162-459d-817a-1913f54ec483-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449151 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449175 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzn9f\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-kube-api-access-fzn9f\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449201 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449284 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449318 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449463 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449486 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55lsj\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-kube-api-access-55lsj\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449510 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449540 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.457543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.449566 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.468627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.468597 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8"] Oct 29 22:44:19.469449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.469417 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" containerID="cri-o://67e91b0aa190fa5515a64739caffeb6b051e2c411aac584cbffd649dfcc2f783" gracePeriod=30 Oct 29 22:44:19.536285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.536253 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj"] Oct 29 22:44:19.536553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.536523 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" containerID="cri-o://9e6edcae2aca1bea380218593365dae7e114fc6de6e09e28347de4fbcd16e841" gracePeriod=30 Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553239 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a2e42-7162-459d-817a-1913f54ec483-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553291 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553317 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzn9f\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-kube-api-access-fzn9f\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553346 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553439 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553472 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553560 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553586 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55lsj\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-kube-api-access-55lsj\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553612 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553644 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553672 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553732 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553775 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553802 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553848 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553874 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553961 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.553993 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554017 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76lj\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-kube-api-access-k76lj\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554041 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f653887-6403-42ab-96ee-dfc750ab4be3-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.554091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554101 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554129 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a2e42-7162-459d-817a-1913f54ec483-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.554943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554153 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.554943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554213 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554244 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.554943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554267 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.554943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.554295 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.556442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.556167 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f653887-6403-42ab-96ee-dfc750ab4be3-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.558976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.558508 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.559836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.559239 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.559836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.559572 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.559836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.559754 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.560154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.560130 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.560537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.560511 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.561386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.561359 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.561855 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.561830 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.561994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.561974 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.562161 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.562077 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.562229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.562200 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-credential-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.564096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.562453 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.564096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.562879 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-certs\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.564096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.563577 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.564096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.563836 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-socket\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.564096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.563961 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a2e42-7162-459d-817a-1913f54ec483-istiod-ca-cert\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.564541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.564485 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-data\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.565149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.565131 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp"] Oct 29 22:44:19.565349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.565321 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9"] Oct 29 22:44:19.566040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.565999 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.573572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.573518 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a2e42-7162-459d-817a-1913f54ec483-istio-podinfo\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.577136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.577103 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-envoy\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.580201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.580146 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.582235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.582213 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzn9f\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-kube-api-access-fzn9f\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.588477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.588357 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.590629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.590605 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55lsj\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-kube-api-access-55lsj\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.591316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.591286 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76lj\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-kube-api-access-k76lj\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.592226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.592203 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-token\") pod \"e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.670482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.670423 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:19.806494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.806348 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:19.824351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.824323 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:19.865183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:19.865100 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:20.034445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.031034 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp"] Oct 29 22:44:20.075127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.075056 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:20.076081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.076033 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.076198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.076162 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.076268 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.076209 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.344290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.344256 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf"] Oct 29 22:44:20.347216 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:44:20.347188 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1a2e42_7162_459d_817a_1913f54ec483.slice/crio-91e4bb544d5891e74673088f5d19e896877773a948c5bb8f387a87d7f19e61d4 WatchSource:0}: Error finding container 91e4bb544d5891e74673088f5d19e896877773a948c5bb8f387a87d7f19e61d4: Status 404 returned error can't find the container with id 91e4bb544d5891e74673088f5d19e896877773a948c5bb8f387a87d7f19e61d4 Oct 29 22:44:20.350997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.350938 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.351085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.351013 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.351085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.351043 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.359597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.358187 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:20.389106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.388528 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9"] Oct 29 22:44:20.392875 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:44:20.392841 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1dce89_199e_4206_b6c6_d3fb049f64e8.slice/crio-d604c217e2636695532fd092ec4d75459bd16be6d60ed646f96fca05c53beb31 WatchSource:0}: Error finding container d604c217e2636695532fd092ec4d75459bd16be6d60ed646f96fca05c53beb31: Status 404 returned error can't find the container with id d604c217e2636695532fd092ec4d75459bd16be6d60ed646f96fca05c53beb31 Oct 29 22:44:20.397931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.397894 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.398028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.398012 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.398076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.398062 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:44:20.417859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.417832 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" event={"ID":"9f1dce89-199e-4206-b6c6-d3fb049f64e8","Type":"ContainerStarted","Data":"d604c217e2636695532fd092ec4d75459bd16be6d60ed646f96fca05c53beb31"} Oct 29 22:44:20.419868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.419843 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" event={"ID":"ab1a2e42-7162-459d-817a-1913f54ec483","Type":"ContainerStarted","Data":"91e4bb544d5891e74673088f5d19e896877773a948c5bb8f387a87d7f19e61d4"} Oct 29 22:44:20.423009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.422976 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" event={"ID":"9f653887-6403-42ab-96ee-dfc750ab4be3","Type":"ContainerStarted","Data":"7d7831d05b955bd38ef7c650ee08d29d6b4ae637f45d6cec2c036a3d08093be9"} Oct 29 22:44:20.423168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.423151 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" event={"ID":"9f653887-6403-42ab-96ee-dfc750ab4be3","Type":"ContainerStarted","Data":"753a248397b37b96e5094200c698dc723262105027f6c98139f6330dbe88feba"} Oct 29 22:44:20.536426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.506723 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podStartSLOduration=2.506702251 podStartE2EDuration="2.506702251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:20.499945497 +0000 UTC m=+1722.981262991" watchObservedRunningTime="2026-10-29 22:44:20.506702251 +0000 UTC m=+1722.988019745" Oct 29 22:44:20.670653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.670610 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:20.955840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.955766 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:20.976234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.970681 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9"] Oct 29 22:44:20.976234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:20.970929 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" containerID="cri-o://37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428" gracePeriod=30 Oct 29 22:44:21.156791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.155780 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:21.433067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.433021 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" event={"ID":"9f1dce89-199e-4206-b6c6-d3fb049f64e8","Type":"ContainerStarted","Data":"048cb6c6e3e403f43aa3d061a42a5cc441941ea537835e2e478db49d86813d0a"} Oct 29 22:44:21.435064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.435033 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" event={"ID":"ab1a2e42-7162-459d-817a-1913f54ec483","Type":"ContainerStarted","Data":"5b8c5a52249f29a93074001dc386a26f7b6f33d672ddc10369f42558f43b74a1"} Oct 29 22:44:21.494185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.494122 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podStartSLOduration=3.494102088 podStartE2EDuration="3.494102088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:21.490000633 +0000 UTC m=+1723.971318127" watchObservedRunningTime="2026-10-29 22:44:21.494102088 +0000 UTC m=+1723.975419582" Oct 29 22:44:21.552424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.550040 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podStartSLOduration=3.550021881 podStartE2EDuration="3.550021881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:21.546480526 +0000 UTC m=+1724.027798020" watchObservedRunningTime="2026-10-29 22:44:21.550021881 +0000 UTC m=+1724.031339375" Oct 29 22:44:21.684465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.678533 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:21.684465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.678597 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:21.859502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.854678 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:21.859502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.854736 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:21.859502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.855060 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:21.859502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:21.855098 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:22.184004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.183969 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:22.184276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.184262 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:22.676915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.676878 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:22.677224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.677194 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:22.838573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.838532 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:22.858549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.858506 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:22.877439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.852940 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:22.894432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.892022 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:22.894432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:22.891954 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:23.163970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.134050 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:23.542907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.542663 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:23.554992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.554944 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:23.708303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.708261 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:23.708528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.708499 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:23.762535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.762485 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:23.824470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.823642 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:23.824470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.823704 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:23.843419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.826014 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:23.843419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:23.826063 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:24.626442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.623596 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:44:24.626442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.624230 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:44:24.714870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.684993 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:24.714870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.685051 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:24.714870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.685435 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:24.738688 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.715593 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:24.828504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.822700 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:44:24.830101 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.829754 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:44:24.891198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.846935 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:24.891198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:24.847012 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:25.099417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.061560 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:25.099417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.061619 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:25.153955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.153914 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:25.528613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.528578 5610 generic.go:356] "Generic (PLEG): container finished" podID="e0301254-4c8b-47b7-9011-c896efee6212" containerID="a946d81c76a0acf46d72e19829074c17c8acd0d26f986b7b84d299d6da1ce00c" exitCode=0 Oct 29 22:44:25.529135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.529100 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" event={"ID":"e0301254-4c8b-47b7-9011-c896efee6212","Type":"ContainerDied","Data":"a946d81c76a0acf46d72e19829074c17c8acd0d26f986b7b84d299d6da1ce00c"} Oct 29 22:44:25.603475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.579249 5610 generic.go:356] "Generic (PLEG): container finished" podID="b8913333-9e24-419a-998c-6e8c604e994e" containerID="9e6edcae2aca1bea380218593365dae7e114fc6de6e09e28347de4fbcd16e841" exitCode=0 Oct 29 22:44:25.603894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.580541 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" event={"ID":"b8913333-9e24-419a-998c-6e8c604e994e","Type":"ContainerDied","Data":"9e6edcae2aca1bea380218593365dae7e114fc6de6e09e28347de4fbcd16e841"} Oct 29 22:44:25.663598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.663544 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:25.687602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.687550 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:25.857904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.805973 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:25.857904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.806047 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:25.857904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.835836 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:25.874793 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.865058 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:25.903563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.875108 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:25.990561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.948182 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:25.990818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.954288 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:25.992596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.978767 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:25.994182 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:25.994139 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:26.378652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:26.378601 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:26.549663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:26.434505 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:26.789422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:26.693985 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:26.789422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:26.694482 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": context deadline exceeded" Oct 29 22:44:26.789422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:26.695537 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:27.012426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.004099 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:27.012426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.004162 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:27.023646 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.023607 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:27.079802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.079756 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:27.080115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.023609 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": context deadline exceeded" Oct 29 22:44:27.289466 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:27.263293 5610 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1d4d6d_d183_4e43_8179_d0e57be42478.slice/crio-conmon-37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428.scope\": RecentStats: unable to find data in memory cache]" Oct 29 22:44:27.568472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.555658 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:27.568472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.555715 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:27.568472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.556781 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:44:27.568472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.557252 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:44:27.642562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.601300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:27.746269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.746230 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:44:27.771938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.771900 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2" podUID="04592e8c-6304-4b3e-8f94-2629a2ff3162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded" Oct 29 22:44:27.772238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.772215 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": context deadline exceeded" Oct 29 22:44:27.777514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.777480 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:44:27.838167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.829018 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:27.838448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.838413 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:27.838691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.838674 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:27.838812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.838793 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:27.900420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.871659 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:27.900420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.871721 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:27.900420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.892578 5610 patch_prober.go:29] interesting pod/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 29 22:44:27.900420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:27.892654 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 29 22:44:28.070503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.070257 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:28.071210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.071178 5610 generic.go:356] "Generic (PLEG): container finished" podID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerID="67e91b0aa190fa5515a64739caffeb6b051e2c411aac584cbffd649dfcc2f783" exitCode=0 Oct 29 22:44:28.142265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.094958 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:28.176120 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:28.151007 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.503s" Oct 29 22:44:28.176120 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.151053 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" event={"ID":"8e132f0f-c890-4055-8967-bbaf48db60d2","Type":"ContainerDied","Data":"67e91b0aa190fa5515a64739caffeb6b051e2c411aac584cbffd649dfcc2f783"} Oct 29 22:44:28.176120 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.151192 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:44:28.176120 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.151350 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:44:28.176120 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.161938 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:44:28.581732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.581680 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:28.739416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.736579 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:44:28.788864 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.742066 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:28.788864 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.742124 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:28.883557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.816453 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:28.883557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.858551 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:28.883557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:28.883191 5610 generic.go:356] "Generic (PLEG): container finished" podID="fce88088-ea36-4392-8466-b74c56c3111d" containerID="294f76ea4243d3f3e262919bc31fddeec97bb235c1ab7b0d1be1e440f989c341" exitCode=0 Oct 29 22:44:29.029821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.029770 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:29.030109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.030076 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:29.030451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.030426 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:44:29.122427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.102520 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": context deadline exceeded" Oct 29 22:44:29.215468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.184516 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:44:29.242996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.242960 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:29.243281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.243250 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:29.647798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.647679 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:44:29.746562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.740437 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:29.890255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:29.890211 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:30.376684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.376630 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-5f78687d5b-zt6f2" podUID="0554501a-13e6-4813-b5d0-b5cb2b14f4c5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.39:50052\" within 1s: context deadline exceeded" Oct 29 22:44:30.377920 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:44:30.377899 5610 logging.go:55] [core] [Channel #3075 SubChannel #3076]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:50052", ServerName: "10.134.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.39:50052: operation was canceled" Oct 29 22:44:30.383031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.382988 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:30.383232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.383053 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:30.383428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.383349 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j" podUID="10fcae09-de89-49fc-98f1-3bdaf39751b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:30.384745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.384708 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 29 22:44:30.384885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.384866 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 29 22:44:30.386639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.386613 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dwclk" podUID="f7c7e838-5a9d-477c-994e-e02cff87ba8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.462538 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-certs\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.462597 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-data\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.462674 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-credential-socket\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.462750 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-kube-api-access-9d69d\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-kube-api-access-9d69d\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.462801 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0301254-4c8b-47b7-9011-c896efee6212-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0301254-4c8b-47b7-9011-c896efee6212-istio-podinfo\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.463073 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-socket\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.463182 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0301254-4c8b-47b7-9011-c896efee6212-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0301254-4c8b-47b7-9011-c896efee6212-istiod-ca-cert\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.463249 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-istio-token\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-istio-token\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.465421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.463369 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-envoy\") pod \"e0301254-4c8b-47b7-9011-c896efee6212\" (UID: \"e0301254-4c8b-47b7-9011-c896efee6212\") " Oct 29 22:44:30.506443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.506186 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:30.547422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.547314 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:30.554487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.551544 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:30.657311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.657257 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-data" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:30.667527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.667489 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-credential-socket" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:30.861476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.824296 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:44:31.033349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.867914 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0301254-4c8b-47b7-9011-c896efee6212-istio-podinfo" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:31.033610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.898288 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:31.256413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.255738 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:31.256413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.887528 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": context deadline exceeded" Oct 29 22:44:31.256413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.869191 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:31.256413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.910668 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-socket" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.910997 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0301254-4c8b-47b7-9011-c896efee6212-istiod-ca-cert" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.914047 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.291197 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.926488 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-kube-api-access-9d69d" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "kube-api-access-9d69d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.971305 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-certs" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.914282 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.918926 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.291610 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-envoy" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.291645 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.949289 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.949501 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 29 22:44:31.292422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:30.868528 5610 generic.go:356] "Generic (PLEG): container finished" podID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerID="37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428" exitCode=0 Oct 29 22:44:31.306083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.305759 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:44:31.308074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.308040 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:44:31.497844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.497746 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:31.546844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.546801 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": context deadline exceeded" Oct 29 22:44:31.547633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.547606 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:44:31.547811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.547789 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:31.547998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.547978 5610 patch_prober.go:29] interesting pod/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:44:31.548130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.548108 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:31.623017 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:31.622977 5610 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.461s" Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.643814 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.644438 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-kube-api-access-4fcbl\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-kube-api-access-4fcbl\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.644497 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-data\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.644559 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-istio-token\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-istio-token\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.644631 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8913333-9e24-419a-998c-6e8c604e994e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8913333-9e24-419a-998c-6e8c604e994e-istio-podinfo\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.644804 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-kube-api-access-bgfrw\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-kube-api-access-bgfrw\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.644849 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-data\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.644883 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-token\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-token\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645005 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-envoy\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645046 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8913333-9e24-419a-998c-6e8c604e994e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8913333-9e24-419a-998c-6e8c604e994e-istiod-ca-cert\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645117 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-credential-socket\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645148 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-certs\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645265 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-socket\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645305 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-certs\") pod \"b8913333-9e24-419a-998c-6e8c604e994e\" (UID: \"b8913333-9e24-419a-998c-6e8c604e994e\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645362 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-podinfo\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645410 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-envoy\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645441 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-socket\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645478 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1d4d6d-d183-4e43-8179-d0e57be42478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1d4d6d-d183-4e43-8179-d0e57be42478-istiod-ca-cert\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.646450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.645541 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-credential-socket\") pod \"3f1d4d6d-d183-4e43-8179-d0e57be42478\" (UID: \"3f1d4d6d-d183-4e43-8179-d0e57be42478\") " Oct 29 22:44:31.647621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.646591 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0301254-4c8b-47b7-9011-c896efee6212-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.647621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.646615 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.647621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.646628 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.647621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.646642 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d69d\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-kube-api-access-9d69d\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.647621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.646666 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0301254-4c8b-47b7-9011-c896efee6212-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.647621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.646680 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0301254-4c8b-47b7-9011-c896efee6212-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.689462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.662036 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.736608 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-istio-token" pod "e0301254-4c8b-47b7-9011-c896efee6212" (UID: "e0301254-4c8b-47b7-9011-c896efee6212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.819753 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.837683 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0301254-4c8b-47b7-9011-c896efee6212-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.841100 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-certs" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.841345 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-socket" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.841560 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-certs" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.905544 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.905600 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.905757 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-data" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.911218 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.911261 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.921856 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-credential-socket" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.922051 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-socket" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.922306 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1d4d6d-d183-4e43-8179-d0e57be42478-istiod-ca-cert" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:31.928819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.926524 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-istio-token" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:31.932742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.931321 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-token" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:31.932742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.931449 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-podinfo" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.954325 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-envoy\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.954575 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e132f0f-c890-4055-8967-bbaf48db60d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e132f0f-c890-4055-8967-bbaf48db60d2-istiod-ca-cert\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.954646 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-credential-socket\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.954760 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-socket\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.954892 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-podinfo\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.954940 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-kube-api-access-gvfn4\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-kube-api-access-gvfn4\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.954977 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-certs\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.955051 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-data\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.955158 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-token\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-token\") pod \"8e132f0f-c890-4055-8967-bbaf48db60d2\" (UID: \"8e132f0f-c890-4055-8967-bbaf48db60d2\") " Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956123 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956150 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956164 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956179 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956193 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956208 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f1d4d6d-d183-4e43-8179-d0e57be42478-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956221 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956236 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956252 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956266 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.956834 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8913333-9e24-419a-998c-6e8c604e994e-istiod-ca-cert" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.957160 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-socket" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.960210 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-kube-api-access-4fcbl" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "kube-api-access-4fcbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.964978 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e132f0f-c890-4055-8967-bbaf48db60d2-istiod-ca-cert" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.965242 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-credential-socket" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.965583 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-certs" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.967827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.966633 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-data" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.977386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.974644 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:44:31.978625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.978590 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-data" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:31.989682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:31.989607 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-envoy" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.057585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.057549 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fce88088-ea36-4392-8466-b74c56c3111d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce88088-ea36-4392-8466-b74c56c3111d-istio-podinfo\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.057752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.057616 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-socket\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.057752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.057667 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-credential-socket\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.057752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.057712 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-data\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.057932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.057825 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce88088-ea36-4392-8466-b74c56c3111d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce88088-ea36-4392-8466-b74c56c3111d-istiod-ca-cert\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.058034 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-certs\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.058078 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-envoy\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.058186 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-kube-api-access-w5sdf\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.058240 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-istio-token\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-istio-token\") pod \"fce88088-ea36-4392-8466-b74c56c3111d\" (UID: \"fce88088-ea36-4392-8466-b74c56c3111d\") " Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059119 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059144 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059158 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059174 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8913333-9e24-419a-998c-6e8c604e994e-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059189 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059202 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059215 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcbl\" (UniqueName: \"kubernetes.io/projected/b8913333-9e24-419a-998c-6e8c604e994e-kube-api-access-4fcbl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059227 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e132f0f-c890-4055-8967-bbaf48db60d2-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.061436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.059244 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.066488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.064905 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-podinfo" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:32.071338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.071137 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-credential-socket" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.072291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.071555 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-socket" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.074184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.074152 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce88088-ea36-4392-8466-b74c56c3111d-istiod-ca-cert" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:32.080832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.074321 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-data" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.097511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.088932 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:44:32.097511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.091098 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-kube-api-access-gvfn4" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "kube-api-access-gvfn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:32.097953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.097927 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-certs" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.105166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.105137 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:44:32.105316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.105213 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:44:32.109821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.109762 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-credential-socket" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.116074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.115962 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-kube-api-access-bgfrw" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "kube-api-access-bgfrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:32.116074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.116044 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8913333-9e24-419a-998c-6e8c604e994e-istio-podinfo" pod "b8913333-9e24-419a-998c-6e8c604e994e" (UID: "b8913333-9e24-419a-998c-6e8c604e994e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:32.130099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.129707 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fce88088-ea36-4392-8466-b74c56c3111d-istio-podinfo" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:32.131086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.131048 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-kube-api-access-w5sdf" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "kube-api-access-w5sdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:32.136248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.136223 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-istio-token" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:32.160362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.160327 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-data\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.160972 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.160948 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-envoy\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.161174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.161158 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-token\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-token\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.161293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.161279 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-credential-socket\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.161476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.161459 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-podinfo\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.161614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.161598 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-kube-api-access-v8l4r\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-kube-api-access-v8l4r\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.161728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.161714 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istiod-ca-cert\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.161856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.161842 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-certs\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.161985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.161970 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-socket\") pod \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\" (UID: \"cfe7e5db-25cf-4299-a4ce-1a7e41598d66\") " Oct 29 22:44:32.185113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.184997 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-socket" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.185430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.185374 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istiod-ca-cert" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:32.193167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.185493 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-envoy" pod "fce88088-ea36-4392-8466-b74c56c3111d" (UID: "fce88088-ea36-4392-8466-b74c56c3111d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.193167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.185950 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-data" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.193167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.188327 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-envoy" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.193167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.189800 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-credential-socket" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.193167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.192786 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-envoy" pod "3f1d4d6d-d183-4e43-8179-d0e57be42478" (UID: "3f1d4d6d-d183-4e43-8179-d0e57be42478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.193167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.192896 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-token" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:32.193638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.193575 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-envoy" pod "8e132f0f-c890-4055-8967-bbaf48db60d2" (UID: "8e132f0f-c890-4055-8967-bbaf48db60d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.198895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.194840 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-podinfo" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:32.198895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.195087 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-token" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:32.198895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.197740 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-certs" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:32.207355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.207206 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-kube-api-access-v8l4r" pod "cfe7e5db-25cf-4299-a4ce-1a7e41598d66" (UID: "cfe7e5db-25cf-4299-a4ce-1a7e41598d66"). InnerVolumeSpecName "kube-api-access-v8l4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:32.279148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.279115 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8913333-9e24-419a-998c-6e8c604e994e-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.282215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.282191 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce88088-ea36-4392-8466-b74c56c3111d-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.282368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.282356 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgfrw\" (UniqueName: \"kubernetes.io/projected/3f1d4d6d-d183-4e43-8179-d0e57be42478-kube-api-access-bgfrw\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.282505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.282487 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.282593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.282584 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.283271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.283250 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvfn4\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-kube-api-access-gvfn4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.288528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.288505 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5sdf\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-kube-api-access-w5sdf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.289652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.289621 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8913333-9e24-419a-998c-6e8c604e994e-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.292049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.291581 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce88088-ea36-4392-8466-b74c56c3111d-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.292049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.291609 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce88088-ea36-4392-8466-b74c56c3111d-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.292049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.291631 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.292049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.291650 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.292049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.291664 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.295930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.295897 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz"] Oct 29 22:44:32.298193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.298168 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:44:32.300999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.298529 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" podUID="8bcc7877-05b4-4d34-b99d-f81c24397ec0" containerName="maas-api" containerID="cri-o://e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3" gracePeriod=30 Oct 29 22:44:32.378704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.300286 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" event={"ID":"fce88088-ea36-4392-8466-b74c56c3111d","Type":"ContainerDied","Data":"294f76ea4243d3f3e262919bc31fddeec97bb235c1ab7b0d1be1e440f989c341"} Oct 29 22:44:32.384849 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.384820 5610 scope.go:122] "RemoveContainer" containerID="294f76ea4243d3f3e262919bc31fddeec97bb235c1ab7b0d1be1e440f989c341" Oct 29 22:44:32.387093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.387070 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:32.387356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.387342 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 22:44:32.387593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.387554 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" event={"ID":"3f1d4d6d-d183-4e43-8179-d0e57be42478","Type":"ContainerDied","Data":"37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428"} Oct 29 22:44:32.395274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.395249 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:44:32.395404 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.395302 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 22:44:32.395404 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.395362 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 22:44:32.395507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.395447 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:44:32.395507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.395499 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 22:44:32.396995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.396975 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:32.399514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.399494 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.400869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.400851 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce88088-ea36-4392-8466-b74c56c3111d-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401006 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.400992 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8l4r\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-kube-api-access-v8l4r\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401092 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401183 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401289 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401411 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401536 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401525 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f1d4d6d-d183-4e43-8179-d0e57be42478-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401618 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401723 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e132f0f-c890-4055-8967-bbaf48db60d2-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401820 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.401924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.401913 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.402024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.402005 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe7e5db-25cf-4299-a4ce-1a7e41598d66-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.407978 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.407417 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:44:32.425383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.425250 5610 generic.go:356] "Generic (PLEG): container finished" podID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerID="aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb" exitCode=0 Oct 29 22:44:32.425383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.425357 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" event={"ID":"cfe7e5db-25cf-4299-a4ce-1a7e41598d66","Type":"ContainerDied","Data":"aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb"} Oct 29 22:44:32.425383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.425450 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" event={"ID":"cfe7e5db-25cf-4299-a4ce-1a7e41598d66","Type":"ContainerDied","Data":"b5de50ae30f20d651f77362719b8e6aba93ec9fa38ad742a7952c4c39a88c03f"} Oct 29 22:44:32.425383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.425366 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m" Oct 29 22:44:32.429722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.429664 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" event={"ID":"e0301254-4c8b-47b7-9011-c896efee6212","Type":"ContainerDied","Data":"d5c62f3a3beda288ac9ed0bda43c1dbab4d95159e51a7560966c1e5d7d08aee1"} Oct 29 22:44:32.429722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.429709 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp" Oct 29 22:44:32.431033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.430953 5610 scope.go:122] "RemoveContainer" containerID="37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428" Oct 29 22:44:32.437004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.436621 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" event={"ID":"8e132f0f-c890-4055-8967-bbaf48db60d2","Type":"ContainerDied","Data":"4ee916ebdda4c842c6ac4b100bacc744be7e8dcdc19121c422883af3879d59df"} Oct 29 22:44:32.437004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.436744 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8" Oct 29 22:44:32.445706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.442645 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" event={"ID":"fce88088-ea36-4392-8466-b74c56c3111d","Type":"ContainerDied","Data":"5e9be5ce0d9f175b3c3bcd05508d2f67b79cfe12ea1bcc6247f8a3847a825412"} Oct 29 22:44:32.445706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.442750 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk" Oct 29 22:44:32.468333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.468292 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" event={"ID":"3f1d4d6d-d183-4e43-8179-d0e57be42478","Type":"ContainerDied","Data":"c6fe159b4ab1a39c13c56cefd4f04f86b8d350126e89b775b738beb9d2faf110"} Oct 29 22:44:32.468547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.468456 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9" Oct 29 22:44:32.480233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.480184 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" event={"ID":"b8913333-9e24-419a-998c-6e8c604e994e","Type":"ContainerDied","Data":"308c76173325a70efbd4cb7bee3eb55154a7247b1badd55b9256799639cd08bc"} Oct 29 22:44:32.488379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.488344 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj" Oct 29 22:44:32.501730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.501701 5610 scope.go:122] "RemoveContainer" containerID="aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb" Oct 29 22:44:32.531357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.531338 5610 scope.go:122] "RemoveContainer" containerID="aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb" Oct 29 22:44:32.532728 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:32.531978 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb\": container with ID starting with aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb not found: ID does not exist" containerID="aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb" Oct 29 22:44:32.532728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.532037 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb"} err="failed to get container status \"aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb\": rpc error: code = NotFound desc = could not find container \"aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb\": container with ID starting with aa49f9b191b6e1db487593f18b07d5b0afef9e14053efe1f9ff84e74f7a627eb not found: ID does not exist" Oct 29 22:44:32.532728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.532065 5610 scope.go:122] "RemoveContainer" containerID="a946d81c76a0acf46d72e19829074c17c8acd0d26f986b7b84d299d6da1ce00c" Oct 29 22:44:32.564791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.564767 5610 scope.go:122] "RemoveContainer" containerID="67e91b0aa190fa5515a64739caffeb6b051e2c411aac584cbffd649dfcc2f783" Oct 29 22:44:32.582818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.582792 5610 scope.go:122] "RemoveContainer" containerID="37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428" Oct 29 22:44:32.584006 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:32.583976 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428\": container with ID starting with 37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428 not found: ID does not exist" containerID="37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428" Oct 29 22:44:32.584109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.584017 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428"} err="failed to get container status \"37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428\": rpc error: code = NotFound desc = could not find container \"37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428\": container with ID starting with 37861841d29be49fd76be3845a11e24463cee8148aa6999d8319b743be17b428 not found: ID does not exist" Oct 29 22:44:32.584109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.584041 5610 scope.go:122] "RemoveContainer" containerID="9e6edcae2aca1bea380218593365dae7e114fc6de6e09e28347de4fbcd16e841" Oct 29 22:44:32.710193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.709361 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:44:32.716517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.716486 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m"] Oct 29 22:44:32.722936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.722908 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-zsz7m"] Oct 29 22:44:32.747140 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.747110 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj"] Oct 29 22:44:32.752004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.751980 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-8wbcj"] Oct 29 22:44:32.800162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.800129 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9"] Oct 29 22:44:32.805591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.805416 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-76rq9"] Oct 29 22:44:32.813371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.812943 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcc7877-05b4-4d34-b99d-f81c24397ec0-kube-api-access-hnvt9\" (UniqueName: \"kubernetes.io/projected/8bcc7877-05b4-4d34-b99d-f81c24397ec0-kube-api-access-hnvt9\") pod \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " Oct 29 22:44:32.813371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.813036 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-maas-api-tls\") pod \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " Oct 29 22:44:32.813371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.813132 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcc7877-05b4-4d34-b99d-f81c24397ec0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc7877-05b4-4d34-b99d-f81c24397ec0-openshift-service-ca\") pod \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " Oct 29 22:44:32.813371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.813181 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-metrics-certs\") pod \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\" (UID: \"8bcc7877-05b4-4d34-b99d-f81c24397ec0\") " Oct 29 22:44:32.813994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.813976 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:32.814622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.814469 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:32.814622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.814512 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcc7877-05b4-4d34-b99d-f81c24397ec0-openshift-service-ca" pod "8bcc7877-05b4-4d34-b99d-f81c24397ec0" (UID: "8bcc7877-05b4-4d34-b99d-f81c24397ec0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:32.815621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.815598 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:44:32.816583 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.816534 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcc7877-05b4-4d34-b99d-f81c24397ec0-kube-api-access-hnvt9" pod "8bcc7877-05b4-4d34-b99d-f81c24397ec0" (UID: "8bcc7877-05b4-4d34-b99d-f81c24397ec0"). InnerVolumeSpecName "kube-api-access-hnvt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:32.820456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.819559 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-metrics-certs" pod "8bcc7877-05b4-4d34-b99d-f81c24397ec0" (UID: "8bcc7877-05b4-4d34-b99d-f81c24397ec0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:32.821523 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.821500 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-maas-api-tls" pod "8bcc7877-05b4-4d34-b99d-f81c24397ec0" (UID: "8bcc7877-05b4-4d34-b99d-f81c24397ec0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:44:32.822383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.822350 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8"] Oct 29 22:44:32.831906 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.831879 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-z47b8"] Oct 29 22:44:32.833303 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.833281 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:32.833878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.833858 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:32.834550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.834533 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:44:32.870446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.864079 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk"] Oct 29 22:44:32.870446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.869248 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-lr5vk"] Oct 29 22:44:32.920149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.914955 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnvt9\" (UniqueName: \"kubernetes.io/projected/8bcc7877-05b4-4d34-b99d-f81c24397ec0-kube-api-access-hnvt9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.920149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.914985 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.920149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.915011 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bcc7877-05b4-4d34-b99d-f81c24397ec0-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.920149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.915028 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bcc7877-05b4-4d34-b99d-f81c24397ec0-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:32.963925 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.963452 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp"] Oct 29 22:44:33.001547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:32.992759 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-433712-openshift-default-594dc9f747-ddrwp"] Oct 29 22:44:33.493283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.493247 5610 generic.go:356] "Generic (PLEG): container finished" podID="8bcc7877-05b4-4d34-b99d-f81c24397ec0" containerID="e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3" exitCode=0 Oct 29 22:44:33.493607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.493318 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" Oct 29 22:44:33.493607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.493332 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" event={"ID":"8bcc7877-05b4-4d34-b99d-f81c24397ec0","Type":"ContainerDied","Data":"e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3"} Oct 29 22:44:33.493607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.493440 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz" event={"ID":"8bcc7877-05b4-4d34-b99d-f81c24397ec0","Type":"ContainerDied","Data":"d97f1bed0c2a37ab6852ca525d4b27a5d402b07e6bc2a3a170f530402ea0515b"} Oct 29 22:44:33.493607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.493481 5610 scope.go:122] "RemoveContainer" containerID="e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3" Oct 29 22:44:33.519023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.518988 5610 scope.go:122] "RemoveContainer" containerID="e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3" Oct 29 22:44:33.519623 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:33.519596 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3\": container with ID starting with e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3 not found: ID does not exist" containerID="e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3" Oct 29 22:44:33.519682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.519632 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3"} err="failed to get container status \"e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3\": rpc error: code = NotFound desc = could not find container \"e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3\": container with ID starting with e3c3f22650ac3d343ce7b1c2037d3aa0d6dda2c83cd9430d4665918c69cae3f3 not found: ID does not exist" Oct 29 22:44:33.549048 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.549015 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz"] Oct 29 22:44:33.552714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.552682 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ae775-6bd4cfffd-6gnsz"] Oct 29 22:44:33.640617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.640581 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" path="/var/lib/kubelet/pods/3f1d4d6d-d183-4e43-8179-d0e57be42478/volumes" Oct 29 22:44:33.641082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.641068 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bcc7877-05b4-4d34-b99d-f81c24397ec0" path="/var/lib/kubelet/pods/8bcc7877-05b4-4d34-b99d-f81c24397ec0/volumes" Oct 29 22:44:33.641564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.641549 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" path="/var/lib/kubelet/pods/8e132f0f-c890-4055-8967-bbaf48db60d2/volumes" Oct 29 22:44:33.642334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.642318 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8913333-9e24-419a-998c-6e8c604e994e" path="/var/lib/kubelet/pods/b8913333-9e24-419a-998c-6e8c604e994e/volumes" Oct 29 22:44:33.642829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.642812 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" path="/var/lib/kubelet/pods/cfe7e5db-25cf-4299-a4ce-1a7e41598d66/volumes" Oct 29 22:44:33.643278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.643264 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0301254-4c8b-47b7-9011-c896efee6212" path="/var/lib/kubelet/pods/e0301254-4c8b-47b7-9011-c896efee6212/volumes" Oct 29 22:44:33.643928 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.643912 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce88088-ea36-4392-8466-b74c56c3111d" path="/var/lib/kubelet/pods/fce88088-ea36-4392-8466-b74c56c3111d/volumes" Oct 29 22:44:33.776640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:33.776491 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:35.780419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:35.774085 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:37.302429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:37.301641 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:37.452010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:37.451976 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:37.656161 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:37.656070 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl"] Oct 29 22:44:37.656345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:37.656317 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" containerID="cri-o://b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2" gracePeriod=30 Oct 29 22:44:37.817912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:37.817872 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:38.015418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:38.014967 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:38.450818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:38.450790 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:39.874099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:39.874057 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:40.012453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:40.012412 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:40.511504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:40.511470 5610 scope.go:122] "RemoveContainer" containerID="281d975ad03562c6b0653f499e8a60a9a872d688f61853f626b03ad9179dc490" Oct 29 22:44:41.734115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:41.734065 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:42.013018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.012935 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp"] Oct 29 22:44:42.013902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013878 5610 cpu_manager.go:414] "containerMap: removing container" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" Oct 29 22:44:42.013902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013899 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013909 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013914 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013925 5610 cpu_manager.go:414] "containerMap: removing container" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013930 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013940 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013945 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013960 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8bcc7877-05b4-4d34-b99d-f81c24397ec0" containerName="maas-api" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013964 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcc7877-05b4-4d34-b99d-f81c24397ec0" containerName="maas-api" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013983 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.013988 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014008 5610 cpu_manager.go:414] "containerMap: removing container" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" Oct 29 22:44:42.014106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014013 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" Oct 29 22:44:42.014774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014117 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e132f0f-c890-4055-8967-bbaf48db60d2" containerName="istio-proxy" Oct 29 22:44:42.014774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014130 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce88088-ea36-4392-8466-b74c56c3111d" containerName="istio-proxy" Oct 29 22:44:42.014774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014140 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0301254-4c8b-47b7-9011-c896efee6212" containerName="istio-proxy" Oct 29 22:44:42.014774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014148 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8913333-9e24-419a-998c-6e8c604e994e" containerName="istio-proxy" Oct 29 22:44:42.014774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014153 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe7e5db-25cf-4299-a4ce-1a7e41598d66" containerName="istio-proxy" Oct 29 22:44:42.014774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014162 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1d4d6d-d183-4e43-8179-d0e57be42478" containerName="istio-proxy" Oct 29 22:44:42.014774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.014170 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcc7877-05b4-4d34-b99d-f81c24397ec0" containerName="maas-api" Oct 29 22:44:42.020345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.020322 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.040075 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.040034 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp"] Oct 29 22:44:42.043269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.043240 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-home\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.043373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.043313 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tls-certs\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.043464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.043435 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfp5\" (UniqueName: \"kubernetes.io/projected/0b1ada16-8a79-4fd4-9320-e38c6f539e97-kube-api-access-xnfp5\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.043507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.043484 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-dshm\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.043548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.043527 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-model-cache\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.043592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.043565 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tmp-dir\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.046366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.046338 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a558b3\"/\"openshift-service-ca.crt\"" Oct 29 22:44:42.046602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.046560 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a558b3\"/\"kube-root-ca.crt\"" Oct 29 22:44:42.048286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.048268 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a558b3\"/\"default-dockercfg-wjbbt\"" Oct 29 22:44:42.048286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.048276 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a558b3\"/\"test-model-a558b3-kserve-self-signed-certs\"" Oct 29 22:44:42.123165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.123130 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:42.145107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145072 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfp5\" (UniqueName: \"kubernetes.io/projected/0b1ada16-8a79-4fd4-9320-e38c6f539e97-kube-api-access-xnfp5\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.145277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145135 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-dshm\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.145277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145183 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-model-cache\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.145277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145202 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tmp-dir\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.145524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145458 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-home\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.145524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145506 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tls-certs\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.145709 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145688 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-model-cache\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.145775 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145704 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tmp-dir\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.145925 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.145892 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-home\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.147932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.147907 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-dshm\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.148065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.148044 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tls-certs\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.164515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.164488 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfp5\" (UniqueName: \"kubernetes.io/projected/0b1ada16-8a79-4fd4-9320-e38c6f539e97-kube-api-access-xnfp5\") pod \"test-model-a558b3-kserve-757d5b65cf-v6zxp\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.343469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.341699 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:42.921026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.920995 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:44:42.933383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:42.933361 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp"] Oct 29 22:44:42.934746 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:44:42.934722 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b1ada16_8a79_4fd4_9320_e38c6f539e97.slice/crio-55d44576d944e2ad43e3adf1fd3a58a3c9e93c9e1200c3897d4744f3c3ef226e WatchSource:0}: Error finding container 55d44576d944e2ad43e3adf1fd3a58a3c9e93c9e1200c3897d4744f3c3ef226e: Status 404 returned error can't find the container with id 55d44576d944e2ad43e3adf1fd3a58a3c9e93c9e1200c3897d4744f3c3ef226e Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034146 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-credential-socket\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034236 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-kube-api-access-8dbvm\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-kube-api-access-8dbvm\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034293 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-certs\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034366 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-podinfo\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034457 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-credential-socket" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034486 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-data\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034508 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-token\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-token\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034538 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istiod-ca-cert\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034573 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-envoy\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034621 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-socket\") pod \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\" (UID: \"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36\") " Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034661 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-certs" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.034975 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-data" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:43.035137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.035041 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istiod-ca-cert" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:44:43.035812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.035227 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-socket" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:43.035812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.035500 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.035812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.035521 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.035812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.035534 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.035812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.035546 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.035812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.035559 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.037701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.037669 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-token" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:43.038290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.038266 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-kube-api-access-8dbvm" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "kube-api-access-8dbvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:44:43.038378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.038275 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-envoy" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:44:43.039085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.039063 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-podinfo" pod "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" (UID: "74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:44:43.137516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.137462 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dbvm\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-kube-api-access-8dbvm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.137516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.137503 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.137516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.137518 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.137753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.137529 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:44:43.602414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.601619 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:43.724559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.724516 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" event={"ID":"0b1ada16-8a79-4fd4-9320-e38c6f539e97","Type":"ContainerStarted","Data":"4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b"} Oct 29 22:44:43.724757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.724572 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" event={"ID":"0b1ada16-8a79-4fd4-9320-e38c6f539e97","Type":"ContainerStarted","Data":"55d44576d944e2ad43e3adf1fd3a58a3c9e93c9e1200c3897d4744f3c3ef226e"} Oct 29 22:44:43.728110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.728080 5610 generic.go:356] "Generic (PLEG): container finished" podID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerID="b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2" exitCode=0 Oct 29 22:44:43.728360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.728339 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" event={"ID":"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36","Type":"ContainerDied","Data":"b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2"} Oct 29 22:44:43.728542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.728526 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" event={"ID":"74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36","Type":"ContainerDied","Data":"3c1136173f78fd8c09ac8770cd8baf483518813b0f6c159d8fed03a68a3b34a8"} Oct 29 22:44:43.728663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.728651 5610 scope.go:122] "RemoveContainer" containerID="b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2" Oct 29 22:44:43.728841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.728424 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl" Oct 29 22:44:43.743049 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:43.743010 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:44:43.744626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.744608 5610 scope.go:122] "RemoveContainer" containerID="157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e" Oct 29 22:44:43.755368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.755347 5610 scope.go:122] "RemoveContainer" containerID="b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2" Oct 29 22:44:43.755774 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:43.755748 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2\": container with ID starting with b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2 not found: ID does not exist" containerID="b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2" Oct 29 22:44:43.755841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.755785 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2"} err="failed to get container status \"b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2\": rpc error: code = NotFound desc = could not find container \"b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2\": container with ID starting with b6d6bc36e09fcd2016175870188d8d913ebe729681366661e834fe23c3c951e2 not found: ID does not exist" Oct 29 22:44:43.755841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.755805 5610 scope.go:122] "RemoveContainer" containerID="157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e" Oct 29 22:44:43.756089 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:43.756075 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e\": container with ID starting with 157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e not found: ID does not exist" containerID="157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e" Oct 29 22:44:43.756146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.756092 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e"} err="failed to get container status \"157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e\": rpc error: code = NotFound desc = could not find container \"157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e\": container with ID starting with 157e6442f451d2d9352d0bf75f81ff6cf1f4e66157c52d341ada59b64b61661e not found: ID does not exist" Oct 29 22:44:43.767060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.766998 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" podStartSLOduration=2.766981811 podStartE2EDuration="2.766981811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:43.764904096 +0000 UTC m=+1746.246221592" watchObservedRunningTime="2026-10-29 22:44:43.766981811 +0000 UTC m=+1746.248299305" Oct 29 22:44:43.805208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.805169 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl"] Oct 29 22:44:43.809857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:43.809834 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1ae775-openshift-default-65f567689c-j6sbl"] Oct 29 22:44:44.342968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:44.342935 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:44.982948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:44.982917 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:45.169339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:45.169302 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:45.638654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:45.638621 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" path="/var/lib/kubelet/pods/74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36/volumes" Oct 29 22:44:46.472685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.472647 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng"] Oct 29 22:44:46.473565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.473543 5610 cpu_manager.go:414] "containerMap: removing container" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" Oct 29 22:44:46.473565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.473561 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" Oct 29 22:44:46.473761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.473586 5610 cpu_manager.go:414] "containerMap: removing container" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" Oct 29 22:44:46.473761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.473591 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" Oct 29 22:44:46.473761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.473689 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" Oct 29 22:44:46.473761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.473705 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dfc1d1-9f28-4ba6-8c38-4a2144e1bb36" containerName="istio-proxy" Oct 29 22:44:46.479262 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.479241 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.498751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.498717 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng"] Oct 29 22:44:46.504829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.504800 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-379a6d-serving-cert\"" Oct 29 22:44:46.599047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.599008 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:46.617555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.617512 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-metrics-certs\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.617741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.617662 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.617790 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.617754 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lwmg\" (UniqueName: \"kubernetes.io/projected/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-kube-api-access-8lwmg\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.617867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.617853 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.725470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.719773 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.725470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.720026 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-metrics-certs\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.725470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.720187 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.725470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.720259 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lwmg\" (UniqueName: \"kubernetes.io/projected/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-kube-api-access-8lwmg\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.725470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.721662 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.725470 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:46.725041 5610 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-serving-cert: secret "maas-api-e2e-inf-a-379a6d-serving-cert" not found Oct 29 22:44:46.725470 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:44:46.725118 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls podName:61f84d72-b1ac-49c9-90d6-9a4735e61e4e nodeName:}" failed. No retries permitted until 2026-10-29 22:44:47.22509692 +0000 UTC m=+1749.706414406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls") pod "maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" (UID: "61f84d72-b1ac-49c9-90d6-9a4735e61e4e") : secret "maas-api-e2e-inf-a-379a6d-serving-cert" not found Oct 29 22:44:46.766648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.758259 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lwmg\" (UniqueName: \"kubernetes.io/projected/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-kube-api-access-8lwmg\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:46.778463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.776300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:46.779692 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:46.779655 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-metrics-certs\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:47.078802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:47.078697 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:47.229058 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:47.229015 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:47.231634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:47.231601 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls\") pod \"maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:47.395091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:47.394671 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:47.773418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:47.753370 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng"] Oct 29 22:44:47.894124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:47.894083 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" event={"ID":"61f84d72-b1ac-49c9-90d6-9a4735e61e4e","Type":"ContainerStarted","Data":"7e6344fb283a9f3d9dcdda8adb22f4a6d3a827bbdd8f4f852dfd85a229e7d5f5"} Oct 29 22:44:48.488026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:48.487985 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:48.905917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:48.905879 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" event={"ID":"61f84d72-b1ac-49c9-90d6-9a4735e61e4e","Type":"ContainerStarted","Data":"6b3081b52f50bb7a462ff821655a0120c72c2633cfc21434d8c8eca848172fa8"} Oct 29 22:44:48.906381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:48.906143 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:48.925570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:48.925543 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:48.950600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:48.950543 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" podStartSLOduration=2.483017278 podStartE2EDuration="2.950528189s" totalImagesPullingTime="467.510911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:48.948427702 +0000 UTC m=+1751.429745192" watchObservedRunningTime="2026-10-29 22:44:48.950528189 +0000 UTC m=+1751.431845683" Oct 29 22:44:49.070351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:49.070310 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:49.264739 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:49.264703 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:49.550468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:49.547976 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:50.436414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:50.435347 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:52.012417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:52.005096 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:52.343091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:52.343009 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:52.355835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:52.355806 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:53.184155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:53.183754 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:44:54.009417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:53.990939 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:55.069427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:55.061526 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:55.139920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:55.139893 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:44:55.482439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:55.482378 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:55.674728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:55.674601 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:57.278017 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:57.277980 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:44:58.991417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:44:58.988692 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:00.281541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.281502 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl"] Oct 29 22:45:00.289518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.289493 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:00.323505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.323471 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl"] Oct 29 22:45:00.328792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.328759 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq"] Oct 29 22:45:00.335115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.335090 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr"] Oct 29 22:45:00.335302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.335213 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:00.343761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.342271 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m"] Oct 29 22:45:00.348836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.346575 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:00.363527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.363498 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr"] Oct 29 22:45:00.363678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.363557 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq"] Oct 29 22:45:00.363678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.363589 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m"] Oct 29 22:45:00.363799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.363683 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:00.380192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.379273 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl\" (UID: \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:00.380192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.379848 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4dlw\" (UniqueName: \"kubernetes.io/projected/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-kube-api-access-c4dlw\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl\" (UID: \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:00.387878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.386258 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 22:45:00.480913 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.480869 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dlw\" (UniqueName: \"kubernetes.io/projected/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-kube-api-access-c4dlw\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl\" (UID: \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:00.481112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.480934 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqsnm\" (UniqueName: \"kubernetes.io/projected/11589eeb-3856-4311-8c4e-e1a61e9f2652-kube-api-access-dqsnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr\" (UID: \"11589eeb-3856-4311-8c4e-e1a61e9f2652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:00.481112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.480961 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11589eeb-3856-4311-8c4e-e1a61e9f2652-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr\" (UID: \"11589eeb-3856-4311-8c4e-e1a61e9f2652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:00.481241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.481118 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl\" (UID: \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:00.481311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.481289 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twktw\" (UniqueName: \"kubernetes.io/projected/9480f1fd-766f-4d7a-be9c-270ea87e806e-kube-api-access-twktw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq\" (UID: \"9480f1fd-766f-4d7a-be9c-270ea87e806e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:00.481456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.481434 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9480f1fd-766f-4d7a-be9c-270ea87e806e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq\" (UID: \"9480f1fd-766f-4d7a-be9c-270ea87e806e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:00.481590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.481573 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ba033-485b-4c3c-9e56-b9752c8c86d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m\" (UID: \"126ba033-485b-4c3c-9e56-b9752c8c86d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:00.481649 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.481606 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj5n2\" (UniqueName: \"kubernetes.io/projected/126ba033-485b-4c3c-9e56-b9752c8c86d7-kube-api-access-zj5n2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m\" (UID: \"126ba033-485b-4c3c-9e56-b9752c8c86d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:00.481708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.481691 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl\" (UID: \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:00.501107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.501078 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dlw\" (UniqueName: \"kubernetes.io/projected/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-kube-api-access-c4dlw\") pod \"maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl\" (UID: \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:00.583347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.583256 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9480f1fd-766f-4d7a-be9c-270ea87e806e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq\" (UID: \"9480f1fd-766f-4d7a-be9c-270ea87e806e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:00.583548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.583357 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ba033-485b-4c3c-9e56-b9752c8c86d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m\" (UID: \"126ba033-485b-4c3c-9e56-b9752c8c86d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:00.583548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.583378 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj5n2\" (UniqueName: \"kubernetes.io/projected/126ba033-485b-4c3c-9e56-b9752c8c86d7-kube-api-access-zj5n2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m\" (UID: \"126ba033-485b-4c3c-9e56-b9752c8c86d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:00.583548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.583496 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqsnm\" (UniqueName: \"kubernetes.io/projected/11589eeb-3856-4311-8c4e-e1a61e9f2652-kube-api-access-dqsnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr\" (UID: \"11589eeb-3856-4311-8c4e-e1a61e9f2652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:00.583548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.583534 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11589eeb-3856-4311-8c4e-e1a61e9f2652-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr\" (UID: \"11589eeb-3856-4311-8c4e-e1a61e9f2652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:00.583783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.583723 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twktw\" (UniqueName: \"kubernetes.io/projected/9480f1fd-766f-4d7a-be9c-270ea87e806e-kube-api-access-twktw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq\" (UID: \"9480f1fd-766f-4d7a-be9c-270ea87e806e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:00.584154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.584130 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ba033-485b-4c3c-9e56-b9752c8c86d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m\" (UID: \"126ba033-485b-4c3c-9e56-b9752c8c86d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:00.584249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.584136 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9480f1fd-766f-4d7a-be9c-270ea87e806e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq\" (UID: \"9480f1fd-766f-4d7a-be9c-270ea87e806e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:00.584364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.584341 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11589eeb-3856-4311-8c4e-e1a61e9f2652-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr\" (UID: \"11589eeb-3856-4311-8c4e-e1a61e9f2652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:00.605938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.605907 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqsnm\" (UniqueName: \"kubernetes.io/projected/11589eeb-3856-4311-8c4e-e1a61e9f2652-kube-api-access-dqsnm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr\" (UID: \"11589eeb-3856-4311-8c4e-e1a61e9f2652\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:00.606069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.606012 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twktw\" (UniqueName: \"kubernetes.io/projected/9480f1fd-766f-4d7a-be9c-270ea87e806e-kube-api-access-twktw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq\" (UID: \"9480f1fd-766f-4d7a-be9c-270ea87e806e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:00.606609 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.606588 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj5n2\" (UniqueName: \"kubernetes.io/projected/126ba033-485b-4c3c-9e56-b9752c8c86d7-kube-api-access-zj5n2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m\" (UID: \"126ba033-485b-4c3c-9e56-b9752c8c86d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:00.626599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.626572 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:00.656826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.656634 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:00.670301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.670052 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:00.684347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.684147 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:00.744049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:00.744017 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:01.070078 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:01.070043 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef95a5a_ef2c_4a05_a225_c8e26c0e9e4c.slice/crio-8f6dcee7779a5ebbe31915313470b646c4a7778254e569f77a4ed2cc34d3b79c WatchSource:0}: Error finding container 8f6dcee7779a5ebbe31915313470b646c4a7778254e569f77a4ed2cc34d3b79c: Status 404 returned error can't find the container with id 8f6dcee7779a5ebbe31915313470b646c4a7778254e569f77a4ed2cc34d3b79c Oct 29 22:45:01.070874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:01.070846 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl"] Oct 29 22:45:01.110378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:01.108859 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr"] Oct 29 22:45:01.110378 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:01.109354 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11589eeb_3856_4311_8c4e_e1a61e9f2652.slice/crio-2e1f3c5745f8948c09a1c8558d8b5c048b3767ddf6f4c614d5c4729b1a2eb2d4 WatchSource:0}: Error finding container 2e1f3c5745f8948c09a1c8558d8b5c048b3767ddf6f4c614d5c4729b1a2eb2d4: Status 404 returned error can't find the container with id 2e1f3c5745f8948c09a1c8558d8b5c048b3767ddf6f4c614d5c4729b1a2eb2d4 Oct 29 22:45:01.113520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:01.113502 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq"] Oct 29 22:45:01.116336 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:01.116307 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9480f1fd_766f_4d7a_be9c_270ea87e806e.slice/crio-5108e999c4768107e248e0e37402c82f04db88a59f92e4bf42cce14e5368bc75 WatchSource:0}: Error finding container 5108e999c4768107e248e0e37402c82f04db88a59f92e4bf42cce14e5368bc75: Status 404 returned error can't find the container with id 5108e999c4768107e248e0e37402c82f04db88a59f92e4bf42cce14e5368bc75 Oct 29 22:45:01.137213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:01.137144 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m"] Oct 29 22:45:01.150894 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:01.150869 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126ba033_485b_4c3c_9e56_b9752c8c86d7.slice/crio-94991c83ca6456e3d38658f51c5c372538ea13c93e58955e46e7212896f17093 WatchSource:0}: Error finding container 94991c83ca6456e3d38658f51c5c372538ea13c93e58955e46e7212896f17093: Status 404 returned error can't find the container with id 94991c83ca6456e3d38658f51c5c372538ea13c93e58955e46e7212896f17093 Oct 29 22:45:01.339184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:01.339136 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" event={"ID":"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c","Type":"ContainerStarted","Data":"8f6dcee7779a5ebbe31915313470b646c4a7778254e569f77a4ed2cc34d3b79c"} Oct 29 22:45:01.340704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:01.340633 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" event={"ID":"9480f1fd-766f-4d7a-be9c-270ea87e806e","Type":"ContainerStarted","Data":"5108e999c4768107e248e0e37402c82f04db88a59f92e4bf42cce14e5368bc75"} Oct 29 22:45:01.341991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:01.341961 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" event={"ID":"126ba033-485b-4c3c-9e56-b9752c8c86d7","Type":"ContainerStarted","Data":"94991c83ca6456e3d38658f51c5c372538ea13c93e58955e46e7212896f17093"} Oct 29 22:45:01.343249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:01.343229 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" event={"ID":"11589eeb-3856-4311-8c4e-e1a61e9f2652","Type":"ContainerStarted","Data":"2e1f3c5745f8948c09a1c8558d8b5c048b3767ddf6f4c614d5c4729b1a2eb2d4"} Oct 29 22:45:02.355072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.355027 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" event={"ID":"126ba033-485b-4c3c-9e56-b9752c8c86d7","Type":"ContainerStarted","Data":"9cacf5b7a096d44eb0a4aa8bc6762459794e1e09ddf61f04152d4a05863d3957"} Oct 29 22:45:02.357061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.357031 5610 generic.go:356] "Generic (PLEG): container finished" podID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerID="7b9f12ac8ade09f1b28712a653a5a1830d3b6dea4ed03d56fae8d97fd946f760" exitCode=60 Oct 29 22:45:02.357195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.357165 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" event={"ID":"11589eeb-3856-4311-8c4e-e1a61e9f2652","Type":"ContainerDied","Data":"7b9f12ac8ade09f1b28712a653a5a1830d3b6dea4ed03d56fae8d97fd946f760"} Oct 29 22:45:02.357380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.357360 5610 scope.go:122] "RemoveContainer" containerID="7b9f12ac8ade09f1b28712a653a5a1830d3b6dea4ed03d56fae8d97fd946f760" Oct 29 22:45:02.362290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.362262 5610 generic.go:356] "Generic (PLEG): container finished" podID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerID="c80b972f263924c1a60ba0ad93eaf24f963c00a1319a41ff09a6c65bb58e6e99" exitCode=60 Oct 29 22:45:02.362473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.362452 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" event={"ID":"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c","Type":"ContainerDied","Data":"c80b972f263924c1a60ba0ad93eaf24f963c00a1319a41ff09a6c65bb58e6e99"} Oct 29 22:45:02.362731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.362713 5610 scope.go:122] "RemoveContainer" containerID="c80b972f263924c1a60ba0ad93eaf24f963c00a1319a41ff09a6c65bb58e6e99" Oct 29 22:45:02.366154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.366124 5610 generic.go:356] "Generic (PLEG): container finished" podID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerID="a7f0b20c6508114aabc1df60d723599067f124b9747fc8b81d32644d240a0ed9" exitCode=60 Oct 29 22:45:02.366286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.366264 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" event={"ID":"9480f1fd-766f-4d7a-be9c-270ea87e806e","Type":"ContainerDied","Data":"a7f0b20c6508114aabc1df60d723599067f124b9747fc8b81d32644d240a0ed9"} Oct 29 22:45:02.366584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.366527 5610 scope.go:122] "RemoveContainer" containerID="a7f0b20c6508114aabc1df60d723599067f124b9747fc8b81d32644d240a0ed9" Oct 29 22:45:02.453136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.444762 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" podStartSLOduration=2.4447395690000002 podStartE2EDuration="2.444739569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:02.418875825 +0000 UTC m=+1764.900193338" watchObservedRunningTime="2026-10-29 22:45:02.444739569 +0000 UTC m=+1764.926057079" Oct 29 22:45:02.482983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:02.482820 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:03.380099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.380053 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" event={"ID":"9480f1fd-766f-4d7a-be9c-270ea87e806e","Type":"ContainerStarted","Data":"684ea396a9168365bc9b838478aa4a3f1766fca14ca7163a66584aaf23350b74"} Oct 29 22:45:03.382110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.382074 5610 generic.go:356] "Generic (PLEG): container finished" podID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerID="9cacf5b7a096d44eb0a4aa8bc6762459794e1e09ddf61f04152d4a05863d3957" exitCode=60 Oct 29 22:45:03.382319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.382109 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" event={"ID":"126ba033-485b-4c3c-9e56-b9752c8c86d7","Type":"ContainerDied","Data":"9cacf5b7a096d44eb0a4aa8bc6762459794e1e09ddf61f04152d4a05863d3957"} Oct 29 22:45:03.382479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.382461 5610 scope.go:122] "RemoveContainer" containerID="9cacf5b7a096d44eb0a4aa8bc6762459794e1e09ddf61f04152d4a05863d3957" Oct 29 22:45:03.385720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.385693 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" event={"ID":"11589eeb-3856-4311-8c4e-e1a61e9f2652","Type":"ContainerStarted","Data":"e695598dc951d0c3887cd6ad15ad15eda23c726bb2f975016a3a12faf380f95f"} Oct 29 22:45:03.388458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.388439 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" event={"ID":"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c","Type":"ContainerStarted","Data":"a942d64dc208f597dc40f631ca88e7576bd5dfbc7d4bc213dc11de6b8c7e6bf3"} Oct 29 22:45:03.389809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.388872 5610 scope.go:122] "RemoveContainer" containerID="a942d64dc208f597dc40f631ca88e7576bd5dfbc7d4bc213dc11de6b8c7e6bf3" Oct 29 22:45:03.389809 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:03.389159 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl_odh-ai-gateway-infra(cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" Oct 29 22:45:03.477946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.477737 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" podStartSLOduration=3.477720295 podStartE2EDuration="3.477720295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.474480761 +0000 UTC m=+1765.955798253" watchObservedRunningTime="2026-10-29 22:45:03.477720295 +0000 UTC m=+1765.959037787" Oct 29 22:45:03.479710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:03.479670 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" podStartSLOduration=3.479656827 podStartE2EDuration="3.479656827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.430428583 +0000 UTC m=+1765.911746075" watchObservedRunningTime="2026-10-29 22:45:03.479656827 +0000 UTC m=+1765.960974365" Oct 29 22:45:04.256680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.256653 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:04.441108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.441077 5610 generic.go:356] "Generic (PLEG): container finished" podID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerID="a942d64dc208f597dc40f631ca88e7576bd5dfbc7d4bc213dc11de6b8c7e6bf3" exitCode=60 Oct 29 22:45:04.441581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.441155 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" event={"ID":"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c","Type":"ContainerDied","Data":"a942d64dc208f597dc40f631ca88e7576bd5dfbc7d4bc213dc11de6b8c7e6bf3"} Oct 29 22:45:04.441581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.441210 5610 scope.go:122] "RemoveContainer" containerID="c80b972f263924c1a60ba0ad93eaf24f963c00a1319a41ff09a6c65bb58e6e99" Oct 29 22:45:04.441581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.441561 5610 scope.go:122] "RemoveContainer" containerID="a942d64dc208f597dc40f631ca88e7576bd5dfbc7d4bc213dc11de6b8c7e6bf3" Oct 29 22:45:04.442028 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:04.442002 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl_odh-ai-gateway-infra(cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" Oct 29 22:45:04.443532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.443510 5610 generic.go:356] "Generic (PLEG): container finished" podID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerID="684ea396a9168365bc9b838478aa4a3f1766fca14ca7163a66584aaf23350b74" exitCode=60 Oct 29 22:45:04.443707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.443692 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" event={"ID":"9480f1fd-766f-4d7a-be9c-270ea87e806e","Type":"ContainerDied","Data":"684ea396a9168365bc9b838478aa4a3f1766fca14ca7163a66584aaf23350b74"} Oct 29 22:45:04.444093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.444076 5610 scope.go:122] "RemoveContainer" containerID="684ea396a9168365bc9b838478aa4a3f1766fca14ca7163a66584aaf23350b74" Oct 29 22:45:04.444457 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:04.444437 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq_odh-ai-gateway-infra(9480f1fd-766f-4d7a-be9c-270ea87e806e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" Oct 29 22:45:04.446494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.446467 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" event={"ID":"126ba033-485b-4c3c-9e56-b9752c8c86d7","Type":"ContainerStarted","Data":"e96ddebd899dc27d91a39b47e26c7a3813930550beddbef1360e053ff78ea3f1"} Oct 29 22:45:04.448602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.448578 5610 generic.go:356] "Generic (PLEG): container finished" podID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerID="e695598dc951d0c3887cd6ad15ad15eda23c726bb2f975016a3a12faf380f95f" exitCode=60 Oct 29 22:45:04.448721 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.448633 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" event={"ID":"11589eeb-3856-4311-8c4e-e1a61e9f2652","Type":"ContainerDied","Data":"e695598dc951d0c3887cd6ad15ad15eda23c726bb2f975016a3a12faf380f95f"} Oct 29 22:45:04.449014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.448992 5610 scope.go:122] "RemoveContainer" containerID="e695598dc951d0c3887cd6ad15ad15eda23c726bb2f975016a3a12faf380f95f" Oct 29 22:45:04.449386 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:04.449364 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr_odh-ai-gateway-infra(11589eeb-3856-4311-8c4e-e1a61e9f2652)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" Oct 29 22:45:04.456027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.456003 5610 scope.go:122] "RemoveContainer" containerID="a7f0b20c6508114aabc1df60d723599067f124b9747fc8b81d32644d240a0ed9" Oct 29 22:45:04.472758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.472657 5610 scope.go:122] "RemoveContainer" containerID="7b9f12ac8ade09f1b28712a653a5a1830d3b6dea4ed03d56fae8d97fd946f760" Oct 29 22:45:04.785912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:04.785815 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:05.467310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:05.467256 5610 scope.go:122] "RemoveContainer" containerID="a942d64dc208f597dc40f631ca88e7576bd5dfbc7d4bc213dc11de6b8c7e6bf3" Oct 29 22:45:05.467939 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:05.467882 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl_odh-ai-gateway-infra(cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" Oct 29 22:45:05.469654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:05.469621 5610 scope.go:122] "RemoveContainer" containerID="684ea396a9168365bc9b838478aa4a3f1766fca14ca7163a66584aaf23350b74" Oct 29 22:45:05.470215 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:05.470180 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq_odh-ai-gateway-infra(9480f1fd-766f-4d7a-be9c-270ea87e806e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" Oct 29 22:45:05.471760 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:05.471729 5610 generic.go:356] "Generic (PLEG): container finished" podID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerID="e96ddebd899dc27d91a39b47e26c7a3813930550beddbef1360e053ff78ea3f1" exitCode=60 Oct 29 22:45:05.471891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:05.471851 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" event={"ID":"126ba033-485b-4c3c-9e56-b9752c8c86d7","Type":"ContainerDied","Data":"e96ddebd899dc27d91a39b47e26c7a3813930550beddbef1360e053ff78ea3f1"} Oct 29 22:45:05.471959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:05.471944 5610 scope.go:122] "RemoveContainer" containerID="9cacf5b7a096d44eb0a4aa8bc6762459794e1e09ddf61f04152d4a05863d3957" Oct 29 22:45:05.472157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:05.472131 5610 scope.go:122] "RemoveContainer" containerID="e96ddebd899dc27d91a39b47e26c7a3813930550beddbef1360e053ff78ea3f1" Oct 29 22:45:05.472592 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:05.472565 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m_odh-ai-gateway-infra(126ba033-485b-4c3c-9e56-b9752c8c86d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" Oct 29 22:45:05.474407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:05.474375 5610 scope.go:122] "RemoveContainer" containerID="e695598dc951d0c3887cd6ad15ad15eda23c726bb2f975016a3a12faf380f95f" Oct 29 22:45:05.474813 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:05.474790 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr_odh-ai-gateway-infra(11589eeb-3856-4311-8c4e-e1a61e9f2652)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" Oct 29 22:45:05.941478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:05.936740 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:06.485208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:06.485180 5610 scope.go:122] "RemoveContainer" containerID="e96ddebd899dc27d91a39b47e26c7a3813930550beddbef1360e053ff78ea3f1" Oct 29 22:45:06.485730 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:06.485593 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m_odh-ai-gateway-infra(126ba033-485b-4c3c-9e56-b9752c8c86d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" Oct 29 22:45:07.594322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:07.594283 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:08.186105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:08.186049 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:08.329234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:08.329183 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:09.142027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.141992 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq"] Oct 29 22:45:09.148217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.148195 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.171763 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.171735 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6bc7b2-serving-cert\"" Oct 29 22:45:09.178466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.178423 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq"] Oct 29 22:45:09.204973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.204510 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-maas-api-tls\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.204973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.204589 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.204973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.204651 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v692l\" (UniqueName: \"kubernetes.io/projected/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-kube-api-access-v692l\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.204973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.204718 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-metrics-certs\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.274590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.274553 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:09.305889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.305858 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-maas-api-tls\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.306086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.305926 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.306086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.305982 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v692l\" (UniqueName: \"kubernetes.io/projected/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-kube-api-access-v692l\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.306086 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.306047 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-metrics-certs\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.307159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.307126 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.308611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.308583 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-maas-api-tls\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.308776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.308760 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-metrics-certs\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.325863 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.325835 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v692l\" (UniqueName: \"kubernetes.io/projected/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-kube-api-access-v692l\") pod \"maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.428133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.428044 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:09.468786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.468749 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:09.818035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:09.818006 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq"] Oct 29 22:45:09.820078 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:09.820048 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcd01ef_3c83_405a_9333_f2bd4d9cf361.slice/crio-a26d420152c33b277bdaf9f9d70da40c50161df9b4cf95ac9065b8f5e4762b2f WatchSource:0}: Error finding container a26d420152c33b277bdaf9f9d70da40c50161df9b4cf95ac9065b8f5e4762b2f: Status 404 returned error can't find the container with id a26d420152c33b277bdaf9f9d70da40c50161df9b4cf95ac9065b8f5e4762b2f Oct 29 22:45:10.560854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:10.557738 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" event={"ID":"1fcd01ef-3c83-405a-9333-f2bd4d9cf361","Type":"ContainerStarted","Data":"41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef"} Oct 29 22:45:10.560854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:10.557798 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" event={"ID":"1fcd01ef-3c83-405a-9333-f2bd4d9cf361","Type":"ContainerStarted","Data":"a26d420152c33b277bdaf9f9d70da40c50161df9b4cf95ac9065b8f5e4762b2f"} Oct 29 22:45:10.560854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:10.558942 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:10.604973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:10.603628 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" podStartSLOduration=1.24360801 podStartE2EDuration="1.603595743s" totalImagesPullingTime="359.987733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:10.597737756 +0000 UTC m=+1773.079055249" watchObservedRunningTime="2026-10-29 22:45:10.603595743 +0000 UTC m=+1773.084913239" Oct 29 22:45:10.788460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:10.788423 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb"] Oct 29 22:45:10.943468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:10.940982 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:11.115384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.111460 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt"] Oct 29 22:45:11.131179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.129759 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.150364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.150332 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-172c10-serving-cert\"" Oct 29 22:45:11.157661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.157618 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt"] Oct 29 22:45:11.287017 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.286915 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f326efd-a274-4b02-98f5-b2ca39ca6209-openshift-service-ca\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.287180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.287080 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgng\" (UniqueName: \"kubernetes.io/projected/1f326efd-a274-4b02-98f5-b2ca39ca6209-kube-api-access-xdgng\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.287256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.287197 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-maas-api-tls\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.287371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.287353 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-metrics-certs\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.324106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.324074 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:11.397465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.388896 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f326efd-a274-4b02-98f5-b2ca39ca6209-openshift-service-ca\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.397465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.389010 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgng\" (UniqueName: \"kubernetes.io/projected/1f326efd-a274-4b02-98f5-b2ca39ca6209-kube-api-access-xdgng\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.397465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.389113 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-maas-api-tls\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.397465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.389260 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-metrics-certs\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.413416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.401099 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f326efd-a274-4b02-98f5-b2ca39ca6209-openshift-service-ca\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.426418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.423723 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-maas-api-tls\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.440416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.437460 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgng\" (UniqueName: \"kubernetes.io/projected/1f326efd-a274-4b02-98f5-b2ca39ca6209-kube-api-access-xdgng\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.456412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.452046 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-metrics-certs\") pod \"maas-api-e2e-inf-a-172c10-576c54476c-twdwt\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.482161 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.481064 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:11.502437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.501577 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:11.848378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:11.848346 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt"] Oct 29 22:45:11.848821 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:11.848529 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f326efd_a274_4b02_98f5_b2ca39ca6209.slice/crio-7de1345a35e25f4ef8c8085e6e7dd1b7c60e2fa7fc7291bddf0b96876c3a5727 WatchSource:0}: Error finding container 7de1345a35e25f4ef8c8085e6e7dd1b7c60e2fa7fc7291bddf0b96876c3a5727: Status 404 returned error can't find the container with id 7de1345a35e25f4ef8c8085e6e7dd1b7c60e2fa7fc7291bddf0b96876c3a5727 Oct 29 22:45:12.024670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:12.024638 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:12.674177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:12.667896 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" event={"ID":"1f326efd-a274-4b02-98f5-b2ca39ca6209","Type":"ContainerStarted","Data":"a26ba01a5b67948abdf36786ba02be43e10d275401abc91d97d6d0aa589d28e5"} Oct 29 22:45:12.674177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:12.667989 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:12.674177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:12.668015 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" event={"ID":"1f326efd-a274-4b02-98f5-b2ca39ca6209","Type":"ContainerStarted","Data":"7de1345a35e25f4ef8c8085e6e7dd1b7c60e2fa7fc7291bddf0b96876c3a5727"} Oct 29 22:45:12.714531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:12.710805 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" podStartSLOduration=1.388344265 podStartE2EDuration="1.710783473s" totalImagesPullingTime="322.439208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:12.709539014 +0000 UTC m=+1775.190856521" watchObservedRunningTime="2026-10-29 22:45:12.710783473 +0000 UTC m=+1775.192100967" Oct 29 22:45:13.053600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:13.053508 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:13.716348 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:13.716305 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:45:14.628616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:14.628584 5610 scope.go:122] "RemoveContainer" containerID="e695598dc951d0c3887cd6ad15ad15eda23c726bb2f975016a3a12faf380f95f" Oct 29 22:45:14.886196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:14.886107 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:15.639152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:15.639121 5610 scope.go:122] "RemoveContainer" containerID="a942d64dc208f597dc40f631ca88e7576bd5dfbc7d4bc213dc11de6b8c7e6bf3" Oct 29 22:45:15.701636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:15.701604 5610 generic.go:356] "Generic (PLEG): container finished" podID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerID="04548e4250d4a0ade30aa5c82c7ef480f95e02cccc2b29e75f942c3552adbaea" exitCode=60 Oct 29 22:45:15.701802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:15.701668 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" event={"ID":"11589eeb-3856-4311-8c4e-e1a61e9f2652","Type":"ContainerDied","Data":"04548e4250d4a0ade30aa5c82c7ef480f95e02cccc2b29e75f942c3552adbaea"} Oct 29 22:45:15.701802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:15.701728 5610 scope.go:122] "RemoveContainer" containerID="e695598dc951d0c3887cd6ad15ad15eda23c726bb2f975016a3a12faf380f95f" Oct 29 22:45:15.702217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:15.702193 5610 scope.go:122] "RemoveContainer" containerID="04548e4250d4a0ade30aa5c82c7ef480f95e02cccc2b29e75f942c3552adbaea" Oct 29 22:45:15.702673 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:15.702645 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr_odh-ai-gateway-infra(11589eeb-3856-4311-8c4e-e1a61e9f2652)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" Oct 29 22:45:16.397707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:16.397626 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:16.713129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:16.713091 5610 generic.go:356] "Generic (PLEG): container finished" podID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerID="3f6ca2f02885ae089415b135514c649f7323c504a7e120674c3280e95c277ca0" exitCode=60 Oct 29 22:45:16.713602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:16.713151 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" event={"ID":"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c","Type":"ContainerDied","Data":"3f6ca2f02885ae089415b135514c649f7323c504a7e120674c3280e95c277ca0"} Oct 29 22:45:16.713602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:16.713196 5610 scope.go:122] "RemoveContainer" containerID="a942d64dc208f597dc40f631ca88e7576bd5dfbc7d4bc213dc11de6b8c7e6bf3" Oct 29 22:45:16.713715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:16.713676 5610 scope.go:122] "RemoveContainer" containerID="3f6ca2f02885ae089415b135514c649f7323c504a7e120674c3280e95c277ca0" Oct 29 22:45:16.714446 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:16.714048 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl_odh-ai-gateway-infra(cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" Oct 29 22:45:16.760270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:16.760240 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr"] Oct 29 22:45:16.952055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:16.952026 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:17.029686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.029643 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11589eeb-3856-4311-8c4e-e1a61e9f2652-kube-api-access-dqsnm\" (UniqueName: \"kubernetes.io/projected/11589eeb-3856-4311-8c4e-e1a61e9f2652-kube-api-access-dqsnm\") pod \"11589eeb-3856-4311-8c4e-e1a61e9f2652\" (UID: \"11589eeb-3856-4311-8c4e-e1a61e9f2652\") " Oct 29 22:45:17.029893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.029872 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11589eeb-3856-4311-8c4e-e1a61e9f2652-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11589eeb-3856-4311-8c4e-e1a61e9f2652-maas-api-service-ca\") pod \"11589eeb-3856-4311-8c4e-e1a61e9f2652\" (UID: \"11589eeb-3856-4311-8c4e-e1a61e9f2652\") " Oct 29 22:45:17.031287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.031255 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11589eeb-3856-4311-8c4e-e1a61e9f2652-maas-api-service-ca" pod "11589eeb-3856-4311-8c4e-e1a61e9f2652" (UID: "11589eeb-3856-4311-8c4e-e1a61e9f2652"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:17.049645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.049617 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11589eeb-3856-4311-8c4e-e1a61e9f2652-kube-api-access-dqsnm" pod "11589eeb-3856-4311-8c4e-e1a61e9f2652" (UID: "11589eeb-3856-4311-8c4e-e1a61e9f2652"). InnerVolumeSpecName "kube-api-access-dqsnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:17.133795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.133761 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqsnm\" (UniqueName: \"kubernetes.io/projected/11589eeb-3856-4311-8c4e-e1a61e9f2652-kube-api-access-dqsnm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:17.133795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.133792 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11589eeb-3856-4311-8c4e-e1a61e9f2652-maas-api-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:17.640245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.639925 5610 scope.go:122] "RemoveContainer" containerID="684ea396a9168365bc9b838478aa4a3f1766fca14ca7163a66584aaf23350b74" Oct 29 22:45:17.676791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.676765 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:45:17.723291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.723265 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" Oct 29 22:45:17.723737 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.723264 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr" event={"ID":"11589eeb-3856-4311-8c4e-e1a61e9f2652","Type":"ContainerDied","Data":"2e1f3c5745f8948c09a1c8558d8b5c048b3767ddf6f4c614d5c4729b1a2eb2d4"} Oct 29 22:45:17.723737 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.723419 5610 scope.go:122] "RemoveContainer" containerID="04548e4250d4a0ade30aa5c82c7ef480f95e02cccc2b29e75f942c3552adbaea" Oct 29 22:45:17.768450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.768382 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl"] Oct 29 22:45:17.800652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.800621 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr"] Oct 29 22:45:17.812865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:17.812817 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-c7b96a-29888565-q7gkr"] Oct 29 22:45:18.038698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.038675 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:18.146803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.146764 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-maas-api-service-ca\") pod \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\" (UID: \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\") " Oct 29 22:45:18.147116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.147091 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-kube-api-access-c4dlw\" (UniqueName: \"kubernetes.io/projected/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-kube-api-access-c4dlw\") pod \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\" (UID: \"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c\") " Oct 29 22:45:18.147887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.147861 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-maas-api-service-ca" pod "cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" (UID: "cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:18.150178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.150140 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-kube-api-access-c4dlw" pod "cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" (UID: "cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c"). InnerVolumeSpecName "kube-api-access-c4dlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:18.219441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.216834 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:18.260498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.250453 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4dlw\" (UniqueName: \"kubernetes.io/projected/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-kube-api-access-c4dlw\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.260498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.250503 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c-maas-api-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:18.628644 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.628564 5610 scope.go:122] "RemoveContainer" containerID="e96ddebd899dc27d91a39b47e26c7a3813930550beddbef1360e053ff78ea3f1" Oct 29 22:45:18.682154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.682125 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:45:18.739003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.738971 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" Oct 29 22:45:18.739511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.739013 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl" event={"ID":"cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c","Type":"ContainerDied","Data":"8f6dcee7779a5ebbe31915313470b646c4a7778254e569f77a4ed2cc34d3b79c"} Oct 29 22:45:18.739511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.739084 5610 scope.go:122] "RemoveContainer" containerID="3f6ca2f02885ae089415b135514c649f7323c504a7e120674c3280e95c277ca0" Oct 29 22:45:18.741359 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.741327 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" event={"ID":"9480f1fd-766f-4d7a-be9c-270ea87e806e","Type":"ContainerStarted","Data":"47273ce47af77179086bb82adbea3e01fb9d5f06aae7c8836bd2cb2a3a967891"} Oct 29 22:45:18.803106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.803073 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl"] Oct 29 22:45:18.816731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:18.816698 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-879596-29888565-95wvl"] Oct 29 22:45:19.398311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.398229 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:19.553606 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.553569 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:19.638490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.638457 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" path="/var/lib/kubelet/pods/11589eeb-3856-4311-8c4e-e1a61e9f2652/volumes" Oct 29 22:45:19.638986 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.638971 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" path="/var/lib/kubelet/pods/cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c/volumes" Oct 29 22:45:19.754929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.754893 5610 generic.go:356] "Generic (PLEG): container finished" podID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerID="25acd2ee7d082fb38be99720cc4d09c4d817f9638c974f023715ec1650f1eda8" exitCode=60 Oct 29 22:45:19.755414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.754966 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" event={"ID":"126ba033-485b-4c3c-9e56-b9752c8c86d7","Type":"ContainerDied","Data":"25acd2ee7d082fb38be99720cc4d09c4d817f9638c974f023715ec1650f1eda8"} Oct 29 22:45:19.755414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.755042 5610 scope.go:122] "RemoveContainer" containerID="e96ddebd899dc27d91a39b47e26c7a3813930550beddbef1360e053ff78ea3f1" Oct 29 22:45:19.755691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.755667 5610 scope.go:122] "RemoveContainer" containerID="25acd2ee7d082fb38be99720cc4d09c4d817f9638c974f023715ec1650f1eda8" Oct 29 22:45:19.756284 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:19.756257 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m_odh-ai-gateway-infra(126ba033-485b-4c3c-9e56-b9752c8c86d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" Oct 29 22:45:19.759848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.759821 5610 generic.go:356] "Generic (PLEG): container finished" podID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerID="47273ce47af77179086bb82adbea3e01fb9d5f06aae7c8836bd2cb2a3a967891" exitCode=60 Oct 29 22:45:19.759996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.759931 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" event={"ID":"9480f1fd-766f-4d7a-be9c-270ea87e806e","Type":"ContainerDied","Data":"47273ce47af77179086bb82adbea3e01fb9d5f06aae7c8836bd2cb2a3a967891"} Oct 29 22:45:19.760379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.760360 5610 scope.go:122] "RemoveContainer" containerID="47273ce47af77179086bb82adbea3e01fb9d5f06aae7c8836bd2cb2a3a967891" Oct 29 22:45:19.760854 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:19.760815 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq_odh-ai-gateway-infra(9480f1fd-766f-4d7a-be9c-270ea87e806e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" Oct 29 22:45:19.774749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.774728 5610 scope.go:122] "RemoveContainer" containerID="684ea396a9168365bc9b838478aa4a3f1766fca14ca7163a66584aaf23350b74" Oct 29 22:45:19.858985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.858957 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d"] Oct 29 22:45:19.859177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.859158 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" containerID="cri-o://e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab" gracePeriod=30 Oct 29 22:45:19.901377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.901338 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq"] Oct 29 22:45:19.952874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:19.952819 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:20.885792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:20.885755 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m"] Oct 29 22:45:21.097263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.097190 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:21.215884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.203694 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9480f1fd-766f-4d7a-be9c-270ea87e806e-kube-api-access-twktw\" (UniqueName: \"kubernetes.io/projected/9480f1fd-766f-4d7a-be9c-270ea87e806e-kube-api-access-twktw\") pod \"9480f1fd-766f-4d7a-be9c-270ea87e806e\" (UID: \"9480f1fd-766f-4d7a-be9c-270ea87e806e\") " Oct 29 22:45:21.257504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.224031 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:21.311860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.303575 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9480f1fd-766f-4d7a-be9c-270ea87e806e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9480f1fd-766f-4d7a-be9c-270ea87e806e-maas-api-service-ca\") pod \"9480f1fd-766f-4d7a-be9c-270ea87e806e\" (UID: \"9480f1fd-766f-4d7a-be9c-270ea87e806e\") " Oct 29 22:45:21.337420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.315529 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9480f1fd-766f-4d7a-be9c-270ea87e806e-maas-api-service-ca" pod "9480f1fd-766f-4d7a-be9c-270ea87e806e" (UID: "9480f1fd-766f-4d7a-be9c-270ea87e806e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.407466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.380752 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:21.412418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.408364 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9480f1fd-766f-4d7a-be9c-270ea87e806e-maas-api-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.426419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.412882 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9480f1fd-766f-4d7a-be9c-270ea87e806e-kube-api-access-twktw" pod "9480f1fd-766f-4d7a-be9c-270ea87e806e" (UID: "9480f1fd-766f-4d7a-be9c-270ea87e806e"). InnerVolumeSpecName "kube-api-access-twktw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.512424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.511526 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126ba033-485b-4c3c-9e56-b9752c8c86d7-kube-api-access-zj5n2\" (UniqueName: \"kubernetes.io/projected/126ba033-485b-4c3c-9e56-b9752c8c86d7-kube-api-access-zj5n2\") pod \"126ba033-485b-4c3c-9e56-b9752c8c86d7\" (UID: \"126ba033-485b-4c3c-9e56-b9752c8c86d7\") " Oct 29 22:45:21.512424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.511588 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126ba033-485b-4c3c-9e56-b9752c8c86d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ba033-485b-4c3c-9e56-b9752c8c86d7-maas-api-service-ca\") pod \"126ba033-485b-4c3c-9e56-b9752c8c86d7\" (UID: \"126ba033-485b-4c3c-9e56-b9752c8c86d7\") " Oct 29 22:45:21.512691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.512536 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twktw\" (UniqueName: \"kubernetes.io/projected/9480f1fd-766f-4d7a-be9c-270ea87e806e-kube-api-access-twktw\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.524670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.513180 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126ba033-485b-4c3c-9e56-b9752c8c86d7-maas-api-service-ca" pod "126ba033-485b-4c3c-9e56-b9752c8c86d7" (UID: "126ba033-485b-4c3c-9e56-b9752c8c86d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:21.527542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.527481 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126ba033-485b-4c3c-9e56-b9752c8c86d7-kube-api-access-zj5n2" pod "126ba033-485b-4c3c-9e56-b9752c8c86d7" (UID: "126ba033-485b-4c3c-9e56-b9752c8c86d7"). InnerVolumeSpecName "kube-api-access-zj5n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:21.614826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.614790 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj5n2\" (UniqueName: \"kubernetes.io/projected/126ba033-485b-4c3c-9e56-b9752c8c86d7-kube-api-access-zj5n2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.614826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.614828 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/126ba033-485b-4c3c-9e56-b9752c8c86d7-maas-api-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:21.801828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.801792 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" Oct 29 22:45:21.802002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.801871 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq" event={"ID":"9480f1fd-766f-4d7a-be9c-270ea87e806e","Type":"ContainerDied","Data":"5108e999c4768107e248e0e37402c82f04db88a59f92e4bf42cce14e5368bc75"} Oct 29 22:45:21.802002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.801934 5610 scope.go:122] "RemoveContainer" containerID="47273ce47af77179086bb82adbea3e01fb9d5f06aae7c8836bd2cb2a3a967891" Oct 29 22:45:21.803950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.803924 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" event={"ID":"126ba033-485b-4c3c-9e56-b9752c8c86d7","Type":"ContainerDied","Data":"94991c83ca6456e3d38658f51c5c372538ea13c93e58955e46e7212896f17093"} Oct 29 22:45:21.804136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.803961 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m" Oct 29 22:45:21.814834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.814813 5610 scope.go:122] "RemoveContainer" containerID="25acd2ee7d082fb38be99720cc4d09c4d817f9638c974f023715ec1650f1eda8" Oct 29 22:45:21.855030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.854989 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq"] Oct 29 22:45:21.894668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.894632 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6bb06a-29888565-v5lxq"] Oct 29 22:45:21.916108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.916080 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m"] Oct 29 22:45:21.922433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:21.922405 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-84304f-29888565-tjx2m"] Oct 29 22:45:23.117320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:23.117286 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:23.246164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:23.246113 5610 patch_prober.go:29] interesting pod/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:45:23.246357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:23.246216 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:45:23.642543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:23.642505 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" path="/var/lib/kubelet/pods/126ba033-485b-4c3c-9e56-b9752c8c86d7/volumes" Oct 29 22:45:23.643199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:23.643173 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" path="/var/lib/kubelet/pods/9480f1fd-766f-4d7a-be9c-270ea87e806e/volumes" Oct 29 22:45:24.305202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:24.305168 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t"] Oct 29 22:45:24.370416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:24.367525 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:25.143839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.128309 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:25.411040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.410960 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:45:25.441588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.441556 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-kube-api-access-gv6nf\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-kube-api-access-gv6nf\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.441802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.441619 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-data\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.441802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.441673 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-podinfo\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.441802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.441734 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-token\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-token\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.441802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.441772 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-socket\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.441802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.441799 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-envoy\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.442066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.441925 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-credential-socket\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.442127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.442074 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64eebe77-a611-4c24-b5a6-a7437eefb088-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eebe77-a611-4c24-b5a6-a7437eefb088-istiod-ca-cert\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.442210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.442184 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-certs\") pod \"64eebe77-a611-4c24-b5a6-a7437eefb088\" (UID: \"64eebe77-a611-4c24-b5a6-a7437eefb088\") " Oct 29 22:45:25.443269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.443243 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-certs" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.443624 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.443598 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-credential-socket" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.443828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.443806 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-socket" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.444429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.444250 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64eebe77-a611-4c24-b5a6-a7437eefb088-istiod-ca-cert" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:25.444819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.444798 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-data" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.446239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.446188 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-token" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.446588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.446538 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-kube-api-access-gv6nf" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "kube-api-access-gv6nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:25.446768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.446743 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-envoy" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:25.447921 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.447886 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-podinfo" pod "64eebe77-a611-4c24-b5a6-a7437eefb088" (UID: "64eebe77-a611-4c24-b5a6-a7437eefb088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:45:25.544938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.544905 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.544938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.544934 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64eebe77-a611-4c24-b5a6-a7437eefb088-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.544938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.544945 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.545251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.544954 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv6nf\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-kube-api-access-gv6nf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.545251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.544962 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.545251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.544972 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.545251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.544981 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.545251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.544991 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.545251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.545001 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64eebe77-a611-4c24-b5a6-a7437eefb088-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:25.845843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.845808 5610 generic.go:356] "Generic (PLEG): container finished" podID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerID="e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab" exitCode=0 Oct 29 22:45:25.846047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.845876 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" Oct 29 22:45:25.846047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.845894 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" event={"ID":"64eebe77-a611-4c24-b5a6-a7437eefb088","Type":"ContainerDied","Data":"e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab"} Oct 29 22:45:25.846047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.845961 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d" event={"ID":"64eebe77-a611-4c24-b5a6-a7437eefb088","Type":"ContainerDied","Data":"bc6780f42c6f14b5654b1a52bb9709e5d45658f2526c5a6cbc5f511d31caf7cc"} Oct 29 22:45:25.846047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.845992 5610 scope.go:122] "RemoveContainer" containerID="e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab" Oct 29 22:45:25.857478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.857458 5610 scope.go:122] "RemoveContainer" containerID="e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab" Oct 29 22:45:25.857773 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:25.857742 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab\": container with ID starting with e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab not found: ID does not exist" containerID="e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab" Oct 29 22:45:25.857886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.857783 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab"} err="failed to get container status \"e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab\": rpc error: code = NotFound desc = could not find container \"e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab\": container with ID starting with e1c7d55cbe46e2b580b270700cd2a01438cc670f97b40f5e5208886013cd1aab not found: ID does not exist" Oct 29 22:45:25.886945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.886904 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" containerID="cri-o://2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f" gracePeriod=45 Oct 29 22:45:25.911580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.911554 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d"] Oct 29 22:45:25.921638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:25.921604 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d478b36e-openshift-default-56b9ccb7bf-zv28d"] Oct 29 22:45:26.252735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.252707 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:45:26.360513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.360477 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-home\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-home\") pod \"4a268be3-9077-44bf-893f-13fe297eea66\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " Oct 29 22:45:26.360708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.360687 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a268be3-9077-44bf-893f-13fe297eea66-kube-api-access-tjn6q\" (UniqueName: \"kubernetes.io/projected/4a268be3-9077-44bf-893f-13fe297eea66-kube-api-access-tjn6q\") pod \"4a268be3-9077-44bf-893f-13fe297eea66\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " Oct 29 22:45:26.360797 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.360766 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a268be3-9077-44bf-893f-13fe297eea66-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a268be3-9077-44bf-893f-13fe297eea66-tls-certs\") pod \"4a268be3-9077-44bf-893f-13fe297eea66\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " Oct 29 22:45:26.360863 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.360834 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-model-cache\") pod \"4a268be3-9077-44bf-893f-13fe297eea66\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " Oct 29 22:45:26.360946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.360930 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-tmp-dir\") pod \"4a268be3-9077-44bf-893f-13fe297eea66\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " Oct 29 22:45:26.361011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.360971 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-dshm\") pod \"4a268be3-9077-44bf-893f-13fe297eea66\" (UID: \"4a268be3-9077-44bf-893f-13fe297eea66\") " Oct 29 22:45:26.398444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.390529 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-home" pod "4a268be3-9077-44bf-893f-13fe297eea66" (UID: "4a268be3-9077-44bf-893f-13fe297eea66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.413472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.404075 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-dshm" pod "4a268be3-9077-44bf-893f-13fe297eea66" (UID: "4a268be3-9077-44bf-893f-13fe297eea66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.413472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.406485 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-model-cache" pod "4a268be3-9077-44bf-893f-13fe297eea66" (UID: "4a268be3-9077-44bf-893f-13fe297eea66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.413472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.411966 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a268be3-9077-44bf-893f-13fe297eea66-kube-api-access-tjn6q" pod "4a268be3-9077-44bf-893f-13fe297eea66" (UID: "4a268be3-9077-44bf-893f-13fe297eea66"). InnerVolumeSpecName "kube-api-access-tjn6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:26.413472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.412482 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-tmp-dir" pod "4a268be3-9077-44bf-893f-13fe297eea66" (UID: "4a268be3-9077-44bf-893f-13fe297eea66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:26.416200 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.416175 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a268be3-9077-44bf-893f-13fe297eea66-tls-certs" pod "4a268be3-9077-44bf-893f-13fe297eea66" (UID: "4a268be3-9077-44bf-893f-13fe297eea66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:26.492512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.491459 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a268be3-9077-44bf-893f-13fe297eea66-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.492512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.491500 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.492512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.491514 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.492512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.491528 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.492512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.491541 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a268be3-9077-44bf-893f-13fe297eea66-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.492512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.491555 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjn6q\" (UniqueName: \"kubernetes.io/projected/4a268be3-9077-44bf-893f-13fe297eea66-kube-api-access-tjn6q\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:26.860451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.860327 5610 generic.go:356] "Generic (PLEG): container finished" podID="4a268be3-9077-44bf-893f-13fe297eea66" containerID="2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f" exitCode=0 Oct 29 22:45:26.860451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.860387 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" Oct 29 22:45:26.860451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.860422 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" event={"ID":"4a268be3-9077-44bf-893f-13fe297eea66","Type":"ContainerDied","Data":"2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f"} Oct 29 22:45:26.860694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.860475 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" event={"ID":"4a268be3-9077-44bf-893f-13fe297eea66","Type":"ContainerDied","Data":"2fcb39c505b9c34cfddcd389cf5f426e52ea5de0821e6437234b917a114efa95"} Oct 29 22:45:26.860694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.860500 5610 scope.go:122] "RemoveContainer" containerID="2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f" Oct 29 22:45:26.873633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.873610 5610 scope.go:122] "RemoveContainer" containerID="2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f" Oct 29 22:45:26.873944 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:26.873922 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f\": container with ID starting with 2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f not found: ID does not exist" containerID="2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f" Oct 29 22:45:26.874013 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.873965 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f"} err="failed to get container status \"2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f\": rpc error: code = NotFound desc = could not find container \"2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f\": container with ID starting with 2e7f6546b6ad6e672395190b8614e8c517b07d4e2fef3897aebdd897120ce71f not found: ID does not exist" Oct 29 22:45:26.912865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.912829 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb"] Oct 29 22:45:26.924128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:26.924095 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb"] Oct 29 22:45:27.090177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:27.090139 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:27.140771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:27.139426 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a26426/test-model-a26426-kserve-9847889bf-l7nmb" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" probeResult="failure" output="Get \"https://10.134.0.141:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:27.638819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:27.638778 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a268be3-9077-44bf-893f-13fe297eea66" path="/var/lib/kubelet/pods/4a268be3-9077-44bf-893f-13fe297eea66/volumes" Oct 29 22:45:27.639547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:27.639525 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" path="/var/lib/kubelet/pods/64eebe77-a611-4c24-b5a6-a7437eefb088/volumes" Oct 29 22:45:28.609309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:28.609263 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:30.163454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:30.157574 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:45:30.163454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:30.157655 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:45:30.244540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:30.244502 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:31.772792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:31.772756 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:31.927239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:31.927189 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:31.937431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:31.937006 5610 generic.go:356] "Generic (PLEG): container finished" podID="c4578473-cc01-462c-920b-c9e76538ca5e" containerID="96c460cc54a909d9fec5f1973122e2a649c28de502549c41c7e4fc1f1bea015e" exitCode=0 Oct 29 22:45:31.937431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:31.937325 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" event={"ID":"c4578473-cc01-462c-920b-c9e76538ca5e","Type":"ContainerDied","Data":"96c460cc54a909d9fec5f1973122e2a649c28de502549c41c7e4fc1f1bea015e"} Oct 29 22:45:31.939109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:31.938417 5610 scope.go:122] "RemoveContainer" containerID="96c460cc54a909d9fec5f1973122e2a649c28de502549c41c7e4fc1f1bea015e" Oct 29 22:45:31.953475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:31.952950 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:31.953475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:31.953042 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:31.953475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:31.953078 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:32.245638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.245605 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:32.618133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.618096 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr"] Oct 29 22:45:32.619131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619104 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:32.619131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619128 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619151 5610 cpu_manager.go:414] "containerMap: removing container" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619157 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619174 5610 cpu_manager.go:414] "containerMap: removing container" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619179 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619189 5610 cpu_manager.go:414] "containerMap: removing container" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619194 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619211 5610 cpu_manager.go:414] "containerMap: removing container" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619216 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619224 5610 cpu_manager.go:414] "containerMap: removing container" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619229 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619238 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619242 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619252 5610 cpu_manager.go:414] "containerMap: removing container" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619258 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619265 5610 cpu_manager.go:414] "containerMap: removing container" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619269 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619279 5610 cpu_manager.go:414] "containerMap: removing container" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619283 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619293 5610 cpu_manager.go:414] "containerMap: removing container" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:32.619294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619299 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619306 5610 cpu_manager.go:414] "containerMap: removing container" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619310 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619320 5610 cpu_manager.go:414] "containerMap: removing container" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619324 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619461 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619472 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619478 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="64eebe77-a611-4c24-b5a6-a7437eefb088" containerName="istio-proxy" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619489 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619495 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a268be3-9077-44bf-893f-13fe297eea66" containerName="main" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619503 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619509 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619515 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619523 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619528 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619535 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.619915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.619540 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="126ba033-485b-4c3c-9e56-b9752c8c86d7" containerName="cleanup" Oct 29 22:45:32.622341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.622324 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.637159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.637091 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ad69b4-openshift-default-dockercfg-mltld\"" Oct 29 22:45:32.656143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.656112 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr"] Oct 29 22:45:32.716585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.716537 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:32.777415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.753630 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgz4\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-kube-api-access-spgz4\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.803415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.785029 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-podinfo\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.803415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.785087 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-data\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.803415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.785149 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-certs\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.803415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.785282 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-credential-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.803415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.785349 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-envoy\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.803415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.785435 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.803415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.785463 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istiod-ca-cert\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.803415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.785529 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-token\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.917469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.895980 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgz4\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-kube-api-access-spgz4\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.917825 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.917797 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-podinfo\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.918007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.917988 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-data\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.918150 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.918137 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-certs\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.918348 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.918335 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-credential-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.918518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.918503 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-envoy\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.918676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.918659 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.918780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.918766 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istiod-ca-cert\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.918921 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.918909 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-token\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.927430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.896162 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:32.953417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.928085 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-credential-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.954131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.951192 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-socket\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.954300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.951537 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-data\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.954426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.951977 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istiod-ca-cert\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.954542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.952332 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-certs\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:32.986414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:32.978285 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-envoy\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:33.010416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.006579 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgz4\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-kube-api-access-spgz4\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:33.038437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.025671 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-podinfo\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:33.038437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.026370 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-token\") pod \"e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:33.055610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.054317 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" event={"ID":"c4578473-cc01-462c-920b-c9e76538ca5e","Type":"ContainerStarted","Data":"953b7988109ab1e62ce2c31794a82e1f3c65980616761594fd2ab7bef142d594"} Oct 29 22:45:33.252057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.252013 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:33.402077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.401965 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:45:33.403423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.403079 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:45:33.403423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.403131 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 29 22:45:33.600812 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:33.600769 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc50dc7_12f1_4a23_bbb0_3c76ac0aea02.slice/crio-17049b1f1254cdca3c942c910bb62d363e04c27e23e3cf2327af80056ea5da62 WatchSource:0}: Error finding container 17049b1f1254cdca3c942c910bb62d363e04c27e23e3cf2327af80056ea5da62: Status 404 returned error can't find the container with id 17049b1f1254cdca3c942c910bb62d363e04c27e23e3cf2327af80056ea5da62 Oct 29 22:45:33.600949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.600835 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr"] Oct 29 22:45:33.603917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.603885 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:33.604027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.603952 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:33.604027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.603982 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:33.785416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:33.784945 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:34.099690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:34.099444 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" event={"ID":"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02","Type":"ContainerStarted","Data":"14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab"} Oct 29 22:45:34.099690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:34.099510 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" event={"ID":"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02","Type":"ContainerStarted","Data":"17049b1f1254cdca3c942c910bb62d363e04c27e23e3cf2327af80056ea5da62"} Oct 29 22:45:34.145736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:34.145679 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podStartSLOduration=2.145663795 podStartE2EDuration="2.145663795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:34.140545596 +0000 UTC m=+1796.621863089" watchObservedRunningTime="2026-10-29 22:45:34.145663795 +0000 UTC m=+1796.626981290" Oct 29 22:45:34.263467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:34.263427 5610 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:45:34.263767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:34.263730 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:45:34.264157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:34.264128 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:34.405474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:34.404558 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:45:34.405474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:34.404623 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 29 22:45:35.212594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:35.212552 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:35.257668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:35.257633 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:35.396418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:35.395061 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:45:35.396418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:35.395118 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 29 22:45:35.447498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:35.444538 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:35.626467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:35.626287 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:36.132967 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:36.132938 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:36.134261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:36.134240 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:45:36.400475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:36.400321 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:45:36.400475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:36.400383 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 29 22:45:36.785038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:36.785002 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:36.921467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:36.921426 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:37.396940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:37.396901 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:45:37.397267 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:37.397226 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 29 22:45:37.490578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:37.490542 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:37.936130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:37.936085 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:38.018133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.018087 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw"] Oct 29 22:45:38.223493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.193466 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:38.223493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.206434 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:38.223493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.206497 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9480f1fd-766f-4d7a-be9c-270ea87e806e" containerName="cleanup" Oct 29 22:45:38.223493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.206806 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef95a5a-ef2c-4a05-a225-c8e26c0e9e4c" containerName="cleanup" Oct 29 22:45:38.223493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.206837 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="11589eeb-3856-4311-8c4e-e1a61e9f2652" containerName="cleanup" Oct 29 22:45:38.453722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.453680 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw"] Oct 29 22:45:38.476505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.459631 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.539545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.537376 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.539626 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krvt\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-kube-api-access-6krvt\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.539698 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.539804 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea5c830-31e9-498e-a774-9c3400e83178-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.539944 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.539972 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea5c830-31e9-498e-a774-9c3400e83178-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.540012 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.540037 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.540073 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.540712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.540687 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-437981-openshift-default-dockercfg-d74jz\"" Oct 29 22:45:38.541278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.541252 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:38.541518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.541492 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:45:38.541627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.541556 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 29 22:45:38.641211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641167 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea5c830-31e9-498e-a774-9c3400e83178-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.641375 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641352 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.641438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641384 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea5c830-31e9-498e-a774-9c3400e83178-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.641476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641459 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.641512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641485 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.641547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641525 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.641683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641663 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.641775 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641758 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krvt\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-kube-api-access-6krvt\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.641856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.641840 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.642092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.642072 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea5c830-31e9-498e-a774-9c3400e83178-istiod-ca-cert\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.642174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.642150 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-data\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.642368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.642348 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.642482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.642381 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-certs\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.642482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.642451 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-credential-socket\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.644376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.644357 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea5c830-31e9-498e-a774-9c3400e83178-istio-podinfo\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.644839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.644813 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-envoy\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.680156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.680121 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krvt\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-kube-api-access-6krvt\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.696535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.696447 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:38.718264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.718231 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-istio-token\") pod \"e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:38.841688 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.841587 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:38.914676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:38.914645 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:39.020174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.020077 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:39.410652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.406689 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" containerID="cri-o://d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505" gracePeriod=45 Oct 29 22:45:39.508515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.501556 5610 patch_prober.go:29] interesting pod/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 29 22:45:39.508515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.501616 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 29 22:45:39.780014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.779843 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw"] Oct 29 22:45:39.788020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.787988 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:39.788202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.788180 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:39.788304 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.788254 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:45:39.845485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.843796 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk"] Oct 29 22:45:39.845948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.845911 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" podUID="10fcd032-fc16-4969-9144-9f030202ca86" containerName="maas-api" containerID="cri-o://267919014d320012d0f6b0d4b1a015ba668e732b08668db183897218a2e88dd9" gracePeriod=30 Oct 29 22:45:39.952516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.952477 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:39.959587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.959559 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:45:39.978036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.977942 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-model-cache\") pod \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " Oct 29 22:45:39.978036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.978035 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tls-certs\") pod \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " Oct 29 22:45:39.978225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.978069 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9b513f-0968-47c2-8201-bd81a45a9e7d-kube-api-access-wx5rb\" (UniqueName: \"kubernetes.io/projected/3e9b513f-0968-47c2-8201-bd81a45a9e7d-kube-api-access-wx5rb\") pod \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " Oct 29 22:45:39.979519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.979216 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-model-cache" pod "3e9b513f-0968-47c2-8201-bd81a45a9e7d" (UID: "3e9b513f-0968-47c2-8201-bd81a45a9e7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:39.983363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.983043 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tls-certs" pod "3e9b513f-0968-47c2-8201-bd81a45a9e7d" (UID: "3e9b513f-0968-47c2-8201-bd81a45a9e7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:39.987054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:39.987028 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9b513f-0968-47c2-8201-bd81a45a9e7d-kube-api-access-wx5rb" pod "3e9b513f-0968-47c2-8201-bd81a45a9e7d" (UID: "3e9b513f-0968-47c2-8201-bd81a45a9e7d"). InnerVolumeSpecName "kube-api-access-wx5rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:40.096717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.085321 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-dshm\") pod \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " Oct 29 22:45:40.096717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.085375 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-home\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-home\") pod \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " Oct 29 22:45:40.098316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.097129 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tmp-dir\") pod \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\" (UID: \"3e9b513f-0968-47c2-8201-bd81a45a9e7d\") " Oct 29 22:45:40.098316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.098043 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:40.098316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.098069 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:40.098316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.098085 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx5rb\" (UniqueName: \"kubernetes.io/projected/3e9b513f-0968-47c2-8201-bd81a45a9e7d-kube-api-access-wx5rb\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:40.116467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.104251 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tmp-dir" pod "3e9b513f-0968-47c2-8201-bd81a45a9e7d" (UID: "3e9b513f-0968-47c2-8201-bd81a45a9e7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:40.116869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.116797 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-home" pod "3e9b513f-0968-47c2-8201-bd81a45a9e7d" (UID: "3e9b513f-0968-47c2-8201-bd81a45a9e7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:40.213157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.211576 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:40.213157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.211618 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:40.224425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.217002 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-dshm" pod "3e9b513f-0968-47c2-8201-bd81a45a9e7d" (UID: "3e9b513f-0968-47c2-8201-bd81a45a9e7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:45:40.313652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.313075 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e9b513f-0968-47c2-8201-bd81a45a9e7d-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:40.538506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.531453 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:45:40.538506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.533081 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:45:40.539720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.539695 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:45:40.619776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.619732 5610 generic.go:356] "Generic (PLEG): container finished" podID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerID="d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505" exitCode=0 Oct 29 22:45:40.619940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.619848 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" event={"ID":"3e9b513f-0968-47c2-8201-bd81a45a9e7d","Type":"ContainerDied","Data":"d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505"} Oct 29 22:45:40.619940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.619889 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" event={"ID":"3e9b513f-0968-47c2-8201-bd81a45a9e7d","Type":"ContainerDied","Data":"6dc2d7b9bcd443911e48f138d494c232d946cd1fab8d477f4596e96bbe5a3d3e"} Oct 29 22:45:40.619940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.619916 5610 scope.go:122] "RemoveContainer" containerID="d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505" Oct 29 22:45:40.626442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.622059 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t" Oct 29 22:45:40.678428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.675205 5610 generic.go:356] "Generic (PLEG): container finished" podID="10fcd032-fc16-4969-9144-9f030202ca86" containerID="267919014d320012d0f6b0d4b1a015ba668e732b08668db183897218a2e88dd9" exitCode=0 Oct 29 22:45:40.678428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.675482 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" event={"ID":"10fcd032-fc16-4969-9144-9f030202ca86","Type":"ContainerDied","Data":"267919014d320012d0f6b0d4b1a015ba668e732b08668db183897218a2e88dd9"} Oct 29 22:45:40.748131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.747839 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" event={"ID":"8ea5c830-31e9-498e-a774-9c3400e83178","Type":"ContainerStarted","Data":"6162de3e1c77f15f766b1d4af933e529ff9e8eca9a21d2b6d8cd6c4dd6b7afca"} Oct 29 22:45:40.748131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.747912 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" event={"ID":"8ea5c830-31e9-498e-a774-9c3400e83178","Type":"ContainerStarted","Data":"4580e8d5996fdae461692eed8552aa52e65c528b32ac4b4390ae7532e65479f0"} Oct 29 22:45:40.902844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.896916 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:45:40.923431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.911791 5610 scope.go:122] "RemoveContainer" containerID="d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505" Oct 29 22:45:40.929420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.927006 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:40.929420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:40.927832 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podStartSLOduration=3.927811082 podStartE2EDuration="3.927811082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:40.84843859 +0000 UTC m=+1803.329756077" watchObservedRunningTime="2026-10-29 22:45:40.927811082 +0000 UTC m=+1803.409128576" Oct 29 22:45:41.081036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.054497 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:41.086320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.086235 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fcd032-fc16-4969-9144-9f030202ca86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd032-fc16-4969-9144-9f030202ca86-openshift-service-ca\") pod \"10fcd032-fc16-4969-9144-9f030202ca86\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " Oct 29 22:45:41.086867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.086771 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fcd032-fc16-4969-9144-9f030202ca86-kube-api-access-q5hvw\" (UniqueName: \"kubernetes.io/projected/10fcd032-fc16-4969-9144-9f030202ca86-kube-api-access-q5hvw\") pod \"10fcd032-fc16-4969-9144-9f030202ca86\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " Oct 29 22:45:41.087846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.087683 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls\") pod \"10fcd032-fc16-4969-9144-9f030202ca86\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " Oct 29 22:45:41.087846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.087743 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-metrics-certs\") pod \"10fcd032-fc16-4969-9144-9f030202ca86\" (UID: \"10fcd032-fc16-4969-9144-9f030202ca86\") " Oct 29 22:45:41.090677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.090651 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fcd032-fc16-4969-9144-9f030202ca86-openshift-service-ca" pod "10fcd032-fc16-4969-9144-9f030202ca86" (UID: "10fcd032-fc16-4969-9144-9f030202ca86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:45:41.094973 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:41.094587 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505\": container with ID starting with d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505 not found: ID does not exist" containerID="d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505" Oct 29 22:45:41.094973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.094635 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505"} err="failed to get container status \"d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505\": rpc error: code = NotFound desc = could not find container \"d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505\": container with ID starting with d6d0d81ff55a5d8c7c1663f5a7d594eac567dc34d72fa46515952f71fbb04505 not found: ID does not exist" Oct 29 22:45:41.101813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.101497 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fcd032-fc16-4969-9144-9f030202ca86-kube-api-access-q5hvw" pod "10fcd032-fc16-4969-9144-9f030202ca86" (UID: "10fcd032-fc16-4969-9144-9f030202ca86"). InnerVolumeSpecName "kube-api-access-q5hvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:45:41.106468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.104917 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls" pod "10fcd032-fc16-4969-9144-9f030202ca86" (UID: "10fcd032-fc16-4969-9144-9f030202ca86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:41.159137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.159084 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-metrics-certs" pod "10fcd032-fc16-4969-9144-9f030202ca86" (UID: "10fcd032-fc16-4969-9144-9f030202ca86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:45:41.161155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.161063 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t"] Oct 29 22:45:41.169856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.169676 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e224cd/test-model-e224cd-kserve-56665b74b7-gkl6t"] Oct 29 22:45:41.191731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.191632 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:41.191731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.191664 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10fcd032-fc16-4969-9144-9f030202ca86-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:41.191731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.191682 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10fcd032-fc16-4969-9144-9f030202ca86-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:41.191731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.191699 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5hvw\" (UniqueName: \"kubernetes.io/projected/10fcd032-fc16-4969-9144-9f030202ca86-kube-api-access-q5hvw\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:45:41.227676 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:41.227640 5610 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e9b513f_0968_47c2_8201_bd81a45a9e7d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e9b513f_0968_47c2_8201_bd81a45a9e7d.slice/crio-6dc2d7b9bcd443911e48f138d494c232d946cd1fab8d477f4596e96bbe5a3d3e\": RecentStats: unable to find data in memory cache]" Oct 29 22:45:41.234894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.234861 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:41.469434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.466404 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:41.638146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.638114 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" path="/var/lib/kubelet/pods/3e9b513f-0968-47c2-8201-bd81a45a9e7d/volumes" Oct 29 22:45:41.754602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.754516 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:41.757606 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.757582 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" Oct 29 22:45:41.757606 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.757599 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk" event={"ID":"10fcd032-fc16-4969-9144-9f030202ca86","Type":"ContainerDied","Data":"eb8689a1987291e5fb99ed295cfa6edae5e903b16d064f9fca244bf05aec6c38"} Oct 29 22:45:41.757767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.757662 5610 scope.go:122] "RemoveContainer" containerID="267919014d320012d0f6b0d4b1a015ba668e732b08668db183897218a2e88dd9" Oct 29 22:45:41.811162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.811137 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk"] Oct 29 22:45:41.814745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.814714 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a26426-bb6c7d6cb-8qqpk"] Oct 29 22:45:41.921333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:41.921302 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:42.766231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:42.766194 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:42.767473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:42.767446 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:45:43.260838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:43.260801 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:43.581237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:43.581141 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:43.637883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:43.637853 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fcd032-fc16-4969-9144-9f030202ca86" path="/var/lib/kubelet/pods/10fcd032-fc16-4969-9144-9f030202ca86/volumes" Oct 29 22:45:43.714586 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:45:43.714553 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:45:44.970199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:44.970141 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:45.100584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.100545 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2"] Oct 29 22:45:45.101533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.101515 5610 cpu_manager.go:414] "containerMap: removing container" podUID="10fcd032-fc16-4969-9144-9f030202ca86" containerName="maas-api" Oct 29 22:45:45.101596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.101534 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fcd032-fc16-4969-9144-9f030202ca86" containerName="maas-api" Oct 29 22:45:45.101596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.101551 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" Oct 29 22:45:45.101596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.101556 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" Oct 29 22:45:45.101699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.101663 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fcd032-fc16-4969-9144-9f030202ca86" containerName="maas-api" Oct 29 22:45:45.101699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.101672 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9b513f-0968-47c2-8201-bd81a45a9e7d" containerName="main" Oct 29 22:45:45.106270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.106250 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:45.112378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.112352 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:45.124770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.124750 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2"] Oct 29 22:45:45.131879 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.131860 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-437981\"" Oct 29 22:45:45.131968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.131869 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-437981-dockercfg-stxd5\"" Oct 29 22:45:45.440699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.440577 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57n6z\" (UniqueName: \"kubernetes.io/projected/9ef0069e-4eb3-4789-a080-b9d4920a5140-kube-api-access-57n6z\") pod \"payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2\" (UID: \"9ef0069e-4eb3-4789-a080-b9d4920a5140\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:45.441178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.441146 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef0069e-4eb3-4789-a080-b9d4920a5140-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2\" (UID: \"9ef0069e-4eb3-4789-a080-b9d4920a5140\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:45.538282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.538246 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:45.586598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.544066 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57n6z\" (UniqueName: \"kubernetes.io/projected/9ef0069e-4eb3-4789-a080-b9d4920a5140-kube-api-access-57n6z\") pod \"payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2\" (UID: \"9ef0069e-4eb3-4789-a080-b9d4920a5140\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:45.588107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.588070 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef0069e-4eb3-4789-a080-b9d4920a5140-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2\" (UID: \"9ef0069e-4eb3-4789-a080-b9d4920a5140\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:45.589235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.551896 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv"] Oct 29 22:45:45.590594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.589067 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef0069e-4eb3-4789-a080-b9d4920a5140-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2\" (UID: \"9ef0069e-4eb3-4789-a080-b9d4920a5140\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:45.592728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.591956 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57n6z\" (UniqueName: \"kubernetes.io/projected/9ef0069e-4eb3-4789-a080-b9d4920a5140-kube-api-access-57n6z\") pod \"payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2\" (UID: \"9ef0069e-4eb3-4789-a080-b9d4920a5140\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:45.606139 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.606103 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv"] Oct 29 22:45:45.606316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.606241 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:45.690948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.690846 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv\" (UID: \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:45.690948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.690919 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5qn\" (UniqueName: \"kubernetes.io/projected/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-kube-api-access-cs5qn\") pod \"payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv\" (UID: \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:45.721056 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.721025 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:45.793351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.793187 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv\" (UID: \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:45.793351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.793257 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5qn\" (UniqueName: \"kubernetes.io/projected/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-kube-api-access-cs5qn\") pod \"payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv\" (UID: \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:45.795092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.795061 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv\" (UID: \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:45.812578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.812550 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5qn\" (UniqueName: \"kubernetes.io/projected/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-kube-api-access-cs5qn\") pod \"payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv\" (UID: \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:45.924553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:45.924512 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:46.032547 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:46.032516 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef0069e_4eb3_4789_a080_b9d4920a5140.slice/crio-25cf8e43c8c0905ff2f2f429ec8daa57cd92b0e257ca43f11efe3e77f096e6e3 WatchSource:0}: Error finding container 25cf8e43c8c0905ff2f2f429ec8daa57cd92b0e257ca43f11efe3e77f096e6e3: Status 404 returned error can't find the container with id 25cf8e43c8c0905ff2f2f429ec8daa57cd92b0e257ca43f11efe3e77f096e6e3 Oct 29 22:45:46.032874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.032686 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2"] Oct 29 22:45:46.227311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.227283 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv"] Oct 29 22:45:46.228183 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:45:46.228158 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb67a1af_c1e4_4990_8c95_5a9c11f9f625.slice/crio-b5daa9b090a4dc99956782592c13077e4bb180a9257f8431c1a4a921069057e7 WatchSource:0}: Error finding container b5daa9b090a4dc99956782592c13077e4bb180a9257f8431c1a4a921069057e7: Status 404 returned error can't find the container with id b5daa9b090a4dc99956782592c13077e4bb180a9257f8431c1a4a921069057e7 Oct 29 22:45:46.808946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.808902 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" event={"ID":"9ef0069e-4eb3-4789-a080-b9d4920a5140","Type":"ContainerStarted","Data":"666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f"} Oct 29 22:45:46.809166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.809003 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" event={"ID":"9ef0069e-4eb3-4789-a080-b9d4920a5140","Type":"ContainerStarted","Data":"25cf8e43c8c0905ff2f2f429ec8daa57cd92b0e257ca43f11efe3e77f096e6e3"} Oct 29 22:45:46.809319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.809298 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:46.810890 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.810863 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" event={"ID":"eb67a1af-c1e4-4990-8c95-5a9c11f9f625","Type":"ContainerStarted","Data":"a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645"} Oct 29 22:45:46.811021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.810910 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" event={"ID":"eb67a1af-c1e4-4990-8c95-5a9c11f9f625","Type":"ContainerStarted","Data":"b5daa9b090a4dc99956782592c13077e4bb180a9257f8431c1a4a921069057e7"} Oct 29 22:45:46.811021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.810960 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:46.847655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.847580 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" podStartSLOduration=1.8475623620000001 podStartE2EDuration="1.847562362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:46.846760332 +0000 UTC m=+1809.328077824" watchObservedRunningTime="2026-10-29 22:45:46.847562362 +0000 UTC m=+1809.328879859" Oct 29 22:45:46.889820 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:46.889766 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" podStartSLOduration=1.889751967 podStartE2EDuration="1.889751967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:46.884868873 +0000 UTC m=+1809.366186367" watchObservedRunningTime="2026-10-29 22:45:46.889751967 +0000 UTC m=+1809.371069459" Oct 29 22:45:47.299416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:47.274851 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:47.904033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:47.903998 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:48.048515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:48.048486 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:48.345514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:48.344697 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:49.761415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:49.749871 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:51.818235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:51.818193 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:54.675124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:54.658699 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:56.295698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:56.295607 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:57.823551 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:57.823515 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 22:45:57.824811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:57.824784 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 22:45:58.220413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:58.208540 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:58.744768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:58.744736 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:59.199168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:59.199137 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:59.380964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:59.380927 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv"] Oct 29 22:45:59.420583 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:59.420553 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf"] Oct 29 22:45:59.592974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:59.592886 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:59.732344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:59.732311 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:45:59.924477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:45:59.914910 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:00.142819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:00.096161 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:01.479381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:01.479340 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:01.760475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:01.760369 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:02.198886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:02.198852 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:02.714461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:02.714426 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:02.876004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:02.875971 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:03.544529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:03.544491 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:04.381821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.381772 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" podUID="209974ec-e547-4d58-884b-1d13784bdabf" containerName="payload-processing" containerID="cri-o://28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045" gracePeriod=25 Oct 29 22:46:04.421854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.421807 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" podUID="171987f3-1f45-4938-a66f-702a5b025b7b" containerName="payload-pre-processing" containerID="cri-o://6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0" gracePeriod=25 Oct 29 22:46:04.820690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.820662 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:46:04.931261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.931227 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209974ec-e547-4d58-884b-1d13784bdabf-kube-api-access-h6bjd\" (UniqueName: \"kubernetes.io/projected/209974ec-e547-4d58-884b-1d13784bdabf-kube-api-access-h6bjd\") pod \"209974ec-e547-4d58-884b-1d13784bdabf\" (UID: \"209974ec-e547-4d58-884b-1d13784bdabf\") " Oct 29 22:46:04.931479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.931306 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209974ec-e547-4d58-884b-1d13784bdabf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209974ec-e547-4d58-884b-1d13784bdabf-plugins-config-volume\") pod \"209974ec-e547-4d58-884b-1d13784bdabf\" (UID: \"209974ec-e547-4d58-884b-1d13784bdabf\") " Oct 29 22:46:04.931781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.931752 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209974ec-e547-4d58-884b-1d13784bdabf-plugins-config-volume" pod "209974ec-e547-4d58-884b-1d13784bdabf" (UID: "209974ec-e547-4d58-884b-1d13784bdabf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:04.932006 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.931986 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209974ec-e547-4d58-884b-1d13784bdabf-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:04.933441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.933416 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209974ec-e547-4d58-884b-1d13784bdabf-kube-api-access-h6bjd" pod "209974ec-e547-4d58-884b-1d13784bdabf" (UID: "209974ec-e547-4d58-884b-1d13784bdabf"). InnerVolumeSpecName "kube-api-access-h6bjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:04.937929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:04.937898 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" podUID="171987f3-1f45-4938-a66f-702a5b025b7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:05.033808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.033728 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6bjd\" (UniqueName: \"kubernetes.io/projected/209974ec-e547-4d58-884b-1d13784bdabf-kube-api-access-h6bjd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:05.105181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.105148 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:05.257189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.257152 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:05.371362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.371272 5610 generic.go:356] "Generic (PLEG): container finished" podID="209974ec-e547-4d58-884b-1d13784bdabf" containerID="28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045" exitCode=0 Oct 29 22:46:05.371640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.371374 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" Oct 29 22:46:05.371640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.371504 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" event={"ID":"209974ec-e547-4d58-884b-1d13784bdabf","Type":"ContainerDied","Data":"28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045"} Oct 29 22:46:05.371640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.371590 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv" event={"ID":"209974ec-e547-4d58-884b-1d13784bdabf","Type":"ContainerDied","Data":"d0093981344363782f2213c81b7ae1e6a1619454f313763a186096b0d033a7c6"} Oct 29 22:46:05.371640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.371619 5610 scope.go:122] "RemoveContainer" containerID="28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045" Oct 29 22:46:05.386622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.386599 5610 scope.go:122] "RemoveContainer" containerID="28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045" Oct 29 22:46:05.386962 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:46:05.386929 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045\": container with ID starting with 28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045 not found: ID does not exist" containerID="28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045" Oct 29 22:46:05.387032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.386971 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045"} err="failed to get container status \"28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045\": rpc error: code = NotFound desc = could not find container \"28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045\": container with ID starting with 28e2b8a9b430635d1d1dd715a4d6bbd1db70fe1918b929c00b1f19d267dda045 not found: ID does not exist" Oct 29 22:46:05.426828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.426795 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv"] Oct 29 22:46:05.438516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.438482 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e224cd-5d647796cd-qvbzv"] Oct 29 22:46:05.560123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.560069 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:05.738413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.729717 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209974ec-e547-4d58-884b-1d13784bdabf" path="/var/lib/kubelet/pods/209974ec-e547-4d58-884b-1d13784bdabf/volumes" Oct 29 22:46:05.738413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.730771 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm"] Oct 29 22:46:05.738413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.732553 5610 cpu_manager.go:414] "containerMap: removing container" podUID="209974ec-e547-4d58-884b-1d13784bdabf" containerName="payload-processing" Oct 29 22:46:05.738413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.732575 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209974ec-e547-4d58-884b-1d13784bdabf" containerName="payload-processing" Oct 29 22:46:05.738413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.732821 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="209974ec-e547-4d58-884b-1d13784bdabf" containerName="payload-processing" Oct 29 22:46:05.750742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.750705 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p"] Oct 29 22:46:05.751062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.751039 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.755955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.755928 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm"] Oct 29 22:46:05.756093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.755964 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p"] Oct 29 22:46:05.756093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.756035 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.810443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.808126 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6bc7b2-openshift-default-dockercfg-g8djs\"" Oct 29 22:46:05.852588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.852555 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.852588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.852591 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.852808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.852624 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.852808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.852776 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.852880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.852816 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkq7\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-kube-api-access-jgkq7\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.852920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.852899 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.852958 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.852927 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be592e7c-e303-428e-8f0b-906554d7b96b-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.852993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.852964 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.853077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853062 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.853117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853096 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxfm\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-kube-api-access-zkxfm\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.853199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853179 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.853269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853256 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.853306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853280 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.853341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853314 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa28810-d8d4-4027-b974-fd3b433a46a2-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.853416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853387 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be592e7c-e303-428e-8f0b-906554d7b96b-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.853462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853451 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.853510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853494 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.853546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.853524 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.955478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955437 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955491 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxfm\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-kube-api-access-zkxfm\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955540 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955592 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955615 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.955667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955653 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa28810-d8d4-4027-b974-fd3b433a46a2-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955698 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be592e7c-e303-428e-8f0b-906554d7b96b-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955734 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955758 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955784 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955817 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955835 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955870 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955880 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955903 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955934 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.956507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.955943 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkq7\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-kube-api-access-jgkq7\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.956507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956052 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.956507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956104 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be592e7c-e303-428e-8f0b-906554d7b96b-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.956507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956148 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.956507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956170 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.956507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956505 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.956848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956521 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.956848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956616 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa28810-d8d4-4027-b974-fd3b433a46a2-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.956848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956770 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.956848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956842 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.957173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956863 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.957173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.956870 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be592e7c-e303-428e-8f0b-906554d7b96b-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.958573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.958546 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.958841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.958823 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.958914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.958893 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be592e7c-e303-428e-8f0b-906554d7b96b-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.958967 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.958950 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.975929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.975900 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.976442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.976369 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:05.976656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.976627 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkq7\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-kube-api-access-jgkq7\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:05.980616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:05.980577 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxfm\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-kube-api-access-zkxfm\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:06.096039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.095942 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:06.098103 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.098075 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:06.444522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.444489 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p"] Oct 29 22:46:06.444886 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:46:06.444865 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe592e7c_e303_428e_8f0b_906554d7b96b.slice/crio-28dfbadf137b6465aaad9b65338eb9afc96956d1e510f8b4cbfbd23b689e06ef WatchSource:0}: Error finding container 28dfbadf137b6465aaad9b65338eb9afc96956d1e510f8b4cbfbd23b689e06ef: Status 404 returned error can't find the container with id 28dfbadf137b6465aaad9b65338eb9afc96956d1e510f8b4cbfbd23b689e06ef Oct 29 22:46:06.448527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.448494 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:06.448623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.448561 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:06.448623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.448592 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:06.455211 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:46:06.455172 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa28810_d8d4_4027_b974_fd3b433a46a2.slice/crio-2fce582104a41261e0723e7380fef871e82a9c6b838ee36ab60828ddc328077b WatchSource:0}: Error finding container 2fce582104a41261e0723e7380fef871e82a9c6b838ee36ab60828ddc328077b: Status 404 returned error can't find the container with id 2fce582104a41261e0723e7380fef871e82a9c6b838ee36ab60828ddc328077b Oct 29 22:46:06.456464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.456443 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm"] Oct 29 22:46:06.461672 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.461642 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:06.461795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.461717 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:06.461795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.461764 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:06.806413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.805773 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:06.974673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:06.974638 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:07.153590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:07.144917 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:07.599340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:07.585449 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" event={"ID":"be592e7c-e303-428e-8f0b-906554d7b96b","Type":"ContainerStarted","Data":"d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72"} Oct 29 22:46:07.599340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:07.591377 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" event={"ID":"be592e7c-e303-428e-8f0b-906554d7b96b","Type":"ContainerStarted","Data":"28dfbadf137b6465aaad9b65338eb9afc96956d1e510f8b4cbfbd23b689e06ef"} Oct 29 22:46:07.599340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:07.591430 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" event={"ID":"7aa28810-d8d4-4027-b974-fd3b433a46a2","Type":"ContainerStarted","Data":"93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7"} Oct 29 22:46:07.599340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:07.591455 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" event={"ID":"7aa28810-d8d4-4027-b974-fd3b433a46a2","Type":"ContainerStarted","Data":"2fce582104a41261e0723e7380fef871e82a9c6b838ee36ab60828ddc328077b"} Oct 29 22:46:07.650549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:07.650491 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podStartSLOduration=2.650472061 podStartE2EDuration="2.650472061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:07.650212043 +0000 UTC m=+1830.131529538" watchObservedRunningTime="2026-10-29 22:46:07.650472061 +0000 UTC m=+1830.131789553" Oct 29 22:46:07.752549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:07.752486 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podStartSLOduration=2.752466711 podStartE2EDuration="2.752466711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:07.751801131 +0000 UTC m=+1830.233118625" watchObservedRunningTime="2026-10-29 22:46:07.752466711 +0000 UTC m=+1830.233784205" Oct 29 22:46:07.974459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:07.973908 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:08.111415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.104797 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:08.111415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.107856 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:08.111415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.107985 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:46:08.111415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.108031 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 29 22:46:08.111415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.108375 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 29 22:46:08.111415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.108429 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 29 22:46:08.481569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.478357 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd"] Oct 29 22:46:08.491633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.490767 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.539445 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.539514 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.539682 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48702783-46a4-4dec-9dab-3c33fcb68587-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.539772 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.539847 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.539889 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48702783-46a4-4dec-9dab-3c33fcb68587-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.539925 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.540044 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96s8w\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-kube-api-access-96s8w\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.541655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.540146 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.546497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.546471 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd"] Oct 29 22:46:08.578439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.563685 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-139eb7-openshift-default-dockercfg-zq8wj\"" Oct 29 22:46:08.642189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642145 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.642619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642198 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.642619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642311 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48702783-46a4-4dec-9dab-3c33fcb68587-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.642619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642383 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.642619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642462 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.642619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642486 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48702783-46a4-4dec-9dab-3c33fcb68587-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.642619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642514 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.642822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642625 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96s8w\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-kube-api-access-96s8w\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.642822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.642706 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.644587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.644562 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-certs\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.644914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.644893 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.645510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.645493 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48702783-46a4-4dec-9dab-3c33fcb68587-istiod-ca-cert\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.645867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.645843 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-credential-socket\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.647523 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.647431 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-data\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.653383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.653047 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48702783-46a4-4dec-9dab-3c33fcb68587-istio-podinfo\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.653793 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.653771 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-envoy\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.753516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.750934 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-istio-token\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.753905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.753879 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96s8w\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-kube-api-access-96s8w\") pod \"e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:08.894418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:08.884676 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:09.073580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:09.055006 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:09.118489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:09.108527 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:46:09.118489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:09.108604 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 29 22:46:09.118489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:09.109728 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 29 22:46:09.118489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:09.109767 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 29 22:46:09.273941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:09.273856 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:10.104418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.102463 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 29 22:46:10.104418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.102523 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 29 22:46:10.104418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.102912 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:46:10.104418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.102943 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 29 22:46:10.335701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.335536 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:10.337536 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.337510 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd"] Oct 29 22:46:10.519558 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:46:10.519518 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48702783_46a4_4dec_9dab_3c33fcb68587.slice/crio-899b57920d005189f68f9e9e517ce60be3d136899e76250287711caa6dd69880 WatchSource:0}: Error finding container 899b57920d005189f68f9e9e517ce60be3d136899e76250287711caa6dd69880: Status 404 returned error can't find the container with id 899b57920d005189f68f9e9e517ce60be3d136899e76250287711caa6dd69880 Oct 29 22:46:10.648420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.644527 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:10.673420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.671562 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:10.673420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:10.671646 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:11.109782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:11.109746 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 29 22:46:11.110897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:11.110864 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 29 22:46:11.151488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:11.151259 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" event={"ID":"48702783-46a4-4dec-9dab-3c33fcb68587","Type":"ContainerStarted","Data":"899b57920d005189f68f9e9e517ce60be3d136899e76250287711caa6dd69880"} Oct 29 22:46:11.187431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:11.160583 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:46:11.187431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:11.160730 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 29 22:46:11.730568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:11.727867 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:11.778416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:11.775237 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:46:12.023847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.020113 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:12.071566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.071508 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" podUID="0b1ada16-8a79-4fd4-9320-e38c6f539e97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 29 22:46:12.121062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.120910 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 29 22:46:12.121062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.120981 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 29 22:46:12.122252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.122221 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:46:12.122540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.122502 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 29 22:46:12.242473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.209339 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:46:12.352149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.352007 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" event={"ID":"48702783-46a4-4dec-9dab-3c33fcb68587","Type":"ContainerStarted","Data":"e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054"} Oct 29 22:46:12.434571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.434515 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" podStartSLOduration=4.434501944 podStartE2EDuration="4.434501944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:12.431908054 +0000 UTC m=+1834.913225548" watchObservedRunningTime="2026-10-29 22:46:12.434501944 +0000 UTC m=+1834.915819437" Oct 29 22:46:12.473423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.471039 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:12.709425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.709281 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:12.889150 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.889117 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:12.889570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.889544 5610 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 22:46:12.889753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:12.889730 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 22:46:13.099501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:13.099299 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 29 22:46:13.099501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:13.099360 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 29 22:46:13.103459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:13.102065 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:46:13.103459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:13.102110 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 29 22:46:13.848097 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:46:13.848005 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:46:13.888417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:13.888219 5610 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 22:46:13.888417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:13.888282 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 22:46:14.101908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:14.101824 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 29 22:46:14.104001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:14.103963 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 29 22:46:14.104552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:14.103900 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 29 22:46:14.104783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:14.104749 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 29 22:46:14.907824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:14.900284 5610 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 22:46:14.907824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:14.900343 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 22:46:14.939365 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:14.939319 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" podUID="171987f3-1f45-4938-a66f-702a5b025b7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:46:15.107568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.107527 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:15.108814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.108788 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:15.109361 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.109335 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:46:15.224234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.224196 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:15.224788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.224765 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:15.225849 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.225827 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:46:15.396413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.395686 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:15.885866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.885832 5610 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 22:46:15.886133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:15.886105 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 22:46:16.886417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:16.885840 5610 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 22:46:16.886417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:16.885910 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 22:46:17.194468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:17.193727 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:17.885731 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:17.885684 5610 patch_prober.go:29] interesting pod/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 29 22:46:17.885946 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:17.885776 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 29 22:46:18.943782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:18.943752 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:18.944765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:18.944738 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:18.945733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:18.945710 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:46:19.157929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:19.157895 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:19.323712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:19.323602 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:20.120417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.116977 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:20.299916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.299874 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:20.427953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.427877 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p"] Oct 29 22:46:20.428138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.428116 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" containerID="cri-o://0fe7244e3e5912626afd9048cf31cc54ca188269988f5b3842b9146a477bab36" gracePeriod=30 Oct 29 22:46:20.454119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.454087 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp"] Oct 29 22:46:20.454358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.454328 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" containerID="cri-o://7d7831d05b955bd38ef7c650ee08d29d6b4ae637f45d6cec2c036a3d08093be9" gracePeriod=30 Oct 29 22:46:20.463002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.462968 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9"] Oct 29 22:46:20.463187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.463167 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" containerID="cri-o://048cb6c6e3e403f43aa3d061a42a5cc441941ea537835e2e478db49d86813d0a" gracePeriod=30 Oct 29 22:46:20.466703 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.466682 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf"] Oct 29 22:46:20.466886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.466866 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" containerID="cri-o://5b8c5a52249f29a93074001dc386a26f7b6f33d672ddc10369f42558f43b74a1" gracePeriod=30 Oct 29 22:46:20.596427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:20.588733 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:21.124264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:21.124224 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:21.314020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:21.313979 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:21.354529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:21.354491 5610 patch_prober.go:29] interesting pod/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:46:21.354705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:21.354544 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:46:21.873507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:21.872708 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:22.097434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:22.095539 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:22.881889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:22.881842 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:24.468493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.468459 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc"] Oct 29 22:46:24.472697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.472675 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.511019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.510974 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc"] Oct 29 22:46:24.535251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.535112 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ad69b4-serving-cert\"" Oct 29 22:46:24.600123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.600084 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:24.629447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.620702 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.629447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.620756 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mx4b\" (UniqueName: \"kubernetes.io/projected/2bdc9534-851a-4f75-8094-9d892edea71a-kube-api-access-4mx4b\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.629447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.620885 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc9534-851a-4f75-8094-9d892edea71a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.629447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.621019 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-metrics-certs\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.770771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.735023 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.770771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.735111 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx4b\" (UniqueName: \"kubernetes.io/projected/2bdc9534-851a-4f75-8094-9d892edea71a-kube-api-access-4mx4b\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.770771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.735205 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc9534-851a-4f75-8094-9d892edea71a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.770771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.735430 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-metrics-certs\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.770771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.748542 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc9534-851a-4f75-8094-9d892edea71a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.810481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.799973 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:24.817451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.816075 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mx4b\" (UniqueName: \"kubernetes.io/projected/2bdc9534-851a-4f75-8094-9d892edea71a-kube-api-access-4mx4b\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.817451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.816447 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-metrics-certs\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.823818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.823775 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-maas-api-tls\") pod \"maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.832901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.832695 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:24.879443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.879364 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:46:24.972462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.972415 5610 generic.go:356] "Generic (PLEG): container finished" podID="171987f3-1f45-4938-a66f-702a5b025b7b" containerID="6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0" exitCode=0 Oct 29 22:46:24.972782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.972507 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" Oct 29 22:46:24.973040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.972526 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" event={"ID":"171987f3-1f45-4938-a66f-702a5b025b7b","Type":"ContainerDied","Data":"6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0"} Oct 29 22:46:24.973189 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.973170 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf" event={"ID":"171987f3-1f45-4938-a66f-702a5b025b7b","Type":"ContainerDied","Data":"18b9b98fb5df34b4ea5e674b45dd6ed16809a21da1464ee7a721e81bca650437"} Oct 29 22:46:24.973311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.973215 5610 scope.go:122] "RemoveContainer" containerID="6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0" Oct 29 22:46:24.980568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.980542 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:24.986216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.986194 5610 scope.go:122] "RemoveContainer" containerID="6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0" Oct 29 22:46:24.986529 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:46:24.986505 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0\": container with ID starting with 6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0 not found: ID does not exist" containerID="6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0" Oct 29 22:46:24.986595 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:24.986539 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0"} err="failed to get container status \"6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0\": rpc error: code = NotFound desc = could not find container \"6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0\": container with ID starting with 6331a710e2521818acc7a1d0d1ced508d7bfd2bc98ddca288b04ae933dd1dec0 not found: ID does not exist" Oct 29 22:46:25.045602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.045521 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171987f3-1f45-4938-a66f-702a5b025b7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171987f3-1f45-4938-a66f-702a5b025b7b-plugins-config-volume\") pod \"171987f3-1f45-4938-a66f-702a5b025b7b\" (UID: \"171987f3-1f45-4938-a66f-702a5b025b7b\") " Oct 29 22:46:25.045778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.045761 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171987f3-1f45-4938-a66f-702a5b025b7b-kube-api-access-7t26d\" (UniqueName: \"kubernetes.io/projected/171987f3-1f45-4938-a66f-702a5b025b7b-kube-api-access-7t26d\") pod \"171987f3-1f45-4938-a66f-702a5b025b7b\" (UID: \"171987f3-1f45-4938-a66f-702a5b025b7b\") " Oct 29 22:46:25.045876 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.045852 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171987f3-1f45-4938-a66f-702a5b025b7b-plugins-config-volume" pod "171987f3-1f45-4938-a66f-702a5b025b7b" (UID: "171987f3-1f45-4938-a66f-702a5b025b7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:25.046547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.046526 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171987f3-1f45-4938-a66f-702a5b025b7b-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.047838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.047818 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171987f3-1f45-4938-a66f-702a5b025b7b-kube-api-access-7t26d" pod "171987f3-1f45-4938-a66f-702a5b025b7b" (UID: "171987f3-1f45-4938-a66f-702a5b025b7b"). InnerVolumeSpecName "kube-api-access-7t26d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:25.148321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.148289 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t26d\" (UniqueName: \"kubernetes.io/projected/171987f3-1f45-4938-a66f-702a5b025b7b-kube-api-access-7t26d\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:25.153958 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.153903 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:25.163588 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:46:25.163562 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdc9534_851a_4f75_8094_9d892edea71a.slice/crio-9ef27d952419b0ad1a2285b3dee27a78b176a2ebae1bfdbfbf0f01b64f37a1c2 WatchSource:0}: Error finding container 9ef27d952419b0ad1a2285b3dee27a78b176a2ebae1bfdbfbf0f01b64f37a1c2: Status 404 returned error can't find the container with id 9ef27d952419b0ad1a2285b3dee27a78b176a2ebae1bfdbfbf0f01b64f37a1c2 Oct 29 22:46:25.165071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.165052 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:46:25.166309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.166284 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc"] Oct 29 22:46:25.323712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.323622 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf"] Oct 29 22:46:25.327644 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.327621 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e224cd-5bfbb5879f-fcsrf"] Oct 29 22:46:25.644682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:25.644278 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171987f3-1f45-4938-a66f-702a5b025b7b" path="/var/lib/kubelet/pods/171987f3-1f45-4938-a66f-702a5b025b7b/volumes" Oct 29 22:46:26.053261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.041312 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:26.056045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.055712 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:46:26.083868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.083745 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" event={"ID":"2bdc9534-851a-4f75-8094-9d892edea71a","Type":"ContainerStarted","Data":"2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2"} Oct 29 22:46:26.083868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.083825 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" event={"ID":"2bdc9534-851a-4f75-8094-9d892edea71a","Type":"ContainerStarted","Data":"9ef27d952419b0ad1a2285b3dee27a78b176a2ebae1bfdbfbf0f01b64f37a1c2"} Oct 29 22:46:26.086270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.086249 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:26.089057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.089030 5610 generic.go:356] "Generic (PLEG): container finished" podID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerID="048cb6c6e3e403f43aa3d061a42a5cc441941ea537835e2e478db49d86813d0a" exitCode=0 Oct 29 22:46:26.089191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.089167 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" event={"ID":"9f1dce89-199e-4206-b6c6-d3fb049f64e8","Type":"ContainerDied","Data":"048cb6c6e3e403f43aa3d061a42a5cc441941ea537835e2e478db49d86813d0a"} Oct 29 22:46:26.091038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.091018 5610 generic.go:356] "Generic (PLEG): container finished" podID="147828d4-7a3e-4f50-964f-7132b042aec2" containerID="0fe7244e3e5912626afd9048cf31cc54ca188269988f5b3842b9146a477bab36" exitCode=0 Oct 29 22:46:26.091210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.091196 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" event={"ID":"147828d4-7a3e-4f50-964f-7132b042aec2","Type":"ContainerDied","Data":"0fe7244e3e5912626afd9048cf31cc54ca188269988f5b3842b9146a477bab36"} Oct 29 22:46:26.092933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.092915 5610 generic.go:356] "Generic (PLEG): container finished" podID="ab1a2e42-7162-459d-817a-1913f54ec483" containerID="5b8c5a52249f29a93074001dc386a26f7b6f33d672ddc10369f42558f43b74a1" exitCode=0 Oct 29 22:46:26.093191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.093075 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" event={"ID":"ab1a2e42-7162-459d-817a-1913f54ec483","Type":"ContainerDied","Data":"5b8c5a52249f29a93074001dc386a26f7b6f33d672ddc10369f42558f43b74a1"} Oct 29 22:46:26.095127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.095022 5610 generic.go:356] "Generic (PLEG): container finished" podID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerID="7d7831d05b955bd38ef7c650ee08d29d6b4ae637f45d6cec2c036a3d08093be9" exitCode=0 Oct 29 22:46:26.095127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.095078 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" event={"ID":"9f653887-6403-42ab-96ee-dfc750ab4be3","Type":"ContainerDied","Data":"7d7831d05b955bd38ef7c650ee08d29d6b4ae637f45d6cec2c036a3d08093be9"} Oct 29 22:46:26.095127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.095120 5610 scope.go:122] "RemoveContainer" containerID="7d7831d05b955bd38ef7c650ee08d29d6b4ae637f45d6cec2c036a3d08093be9" Oct 29 22:46:26.095334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.095243 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp" Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133097 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-certs\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133209 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f653887-6403-42ab-96ee-dfc750ab4be3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f653887-6403-42ab-96ee-dfc750ab4be3-istiod-ca-cert\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133277 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-data\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133295 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-envoy\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133319 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-podinfo\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133479 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-credential-socket\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133506 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-socket\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133530 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-token\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-token\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.133626 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-kube-api-access-55lsj\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-kube-api-access-55lsj\") pod \"9f653887-6403-42ab-96ee-dfc750ab4be3\" (UID: \"9f653887-6403-42ab-96ee-dfc750ab4be3\") " Oct 29 22:46:26.135820 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.135360 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-credential-socket" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.136632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.136444 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.137041 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.136769 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-socket" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.138032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.138010 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f653887-6403-42ab-96ee-dfc750ab4be3-istiod-ca-cert" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:26.138500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.138478 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-certs" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.138794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.138776 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-data" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.150830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.150805 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:46:26.155164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.155096 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" podStartSLOduration=1.782469214 podStartE2EDuration="2.155078001s" totalImagesPullingTime="372.608787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:26.147904979 +0000 UTC m=+1848.629222474" watchObservedRunningTime="2026-10-29 22:46:26.155078001 +0000 UTC m=+1848.636395495" Oct 29 22:46:26.157075 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.157045 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-token" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.157198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.157186 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-podinfo" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:26.157282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.157265 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-envoy" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.160134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.160058 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:46:26.166154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.166111 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-kube-api-access-55lsj" pod "9f653887-6403-42ab-96ee-dfc750ab4be3" (UID: "9f653887-6403-42ab-96ee-dfc750ab4be3"). InnerVolumeSpecName "kube-api-access-55lsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.219415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.219334 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:26.229447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.229413 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:46:26.238007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.237972 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-socket\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.238183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238020 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-envoy\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.238183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238050 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-data\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.238183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238089 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-credential-socket\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.238183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238152 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istiod-ca-cert\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.238420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238294 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-data\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.238420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238316 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-socket" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.238420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238337 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-certs\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.238420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238385 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-socket\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.238623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238463 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-certs\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.238684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238641 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-kube-api-access-fzn9f\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-kube-api-access-fzn9f\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.238735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238691 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-certs" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.238789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238725 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-credential-socket" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.238789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238744 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-kube-api-access-z8z9g\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-kube-api-access-z8z9g\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.238898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238814 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istiod-ca-cert" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:26.238898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238816 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-token\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-token\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.238898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238872 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-credential-socket\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.239076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238911 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/147828d4-7a3e-4f50-964f-7132b042aec2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147828d4-7a3e-4f50-964f-7132b042aec2-istio-podinfo\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.239076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238922 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-data" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.239076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.238974 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-podinfo\") pod \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\" (UID: \"9f1dce89-199e-4206-b6c6-d3fb049f64e8\") " Oct 29 22:46:26.239076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.239027 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147828d4-7a3e-4f50-964f-7132b042aec2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147828d4-7a3e-4f50-964f-7132b042aec2-istiod-ca-cert\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.239276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.239075 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-istio-token\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-istio-token\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.239276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.239107 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-envoy\") pod \"147828d4-7a3e-4f50-964f-7132b042aec2\" (UID: \"147828d4-7a3e-4f50-964f-7132b042aec2\") " Oct 29 22:46:26.239276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.239186 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-socket" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.239899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.239870 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147828d4-7a3e-4f50-964f-7132b042aec2-istiod-ca-cert" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:26.240133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240111 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55lsj\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-kube-api-access-55lsj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240141 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240155 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240168 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240182 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f653887-6403-42ab-96ee-dfc750ab4be3-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240195 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240208 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240219 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240232 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147828d4-7a3e-4f50-964f-7132b042aec2-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240245 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f653887-6403-42ab-96ee-dfc750ab4be3-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240259 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240272 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f653887-6403-42ab-96ee-dfc750ab4be3-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240284 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240295 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240307 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.240574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240530 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-certs" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.241369 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.240990 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-data" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.243758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.243732 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-istio-token" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.243963 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.243941 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-credential-socket" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.244909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.244885 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-podinfo" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:26.248726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.248697 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/147828d4-7a3e-4f50-964f-7132b042aec2-istio-podinfo" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:26.248833 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.248719 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-token" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.248833 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.248768 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-kube-api-access-z8z9g" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "kube-api-access-z8z9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.248964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.248943 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-envoy" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.249148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.249111 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-kube-api-access-fzn9f" pod "9f1dce89-199e-4206-b6c6-d3fb049f64e8" (UID: "9f1dce89-199e-4206-b6c6-d3fb049f64e8"). InnerVolumeSpecName "kube-api-access-fzn9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.250384 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.250362 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-envoy" pod "147828d4-7a3e-4f50-964f-7132b042aec2" (UID: "147828d4-7a3e-4f50-964f-7132b042aec2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.416131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.416069 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-credential-socket\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.416420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.416380 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-envoy\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.416673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.416654 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1a2e42-7162-459d-817a-1913f54ec483-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a2e42-7162-459d-817a-1913f54ec483-istiod-ca-cert\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.416843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.416821 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-kube-api-access-k76lj\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-kube-api-access-k76lj\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.417008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.416994 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-istio-token\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.417229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.417211 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab1a2e42-7162-459d-817a-1913f54ec483-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a2e42-7162-459d-817a-1913f54ec483-istio-podinfo\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.417373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.417353 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-certs\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.417538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.417520 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-data\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.417721 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.417693 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-socket\") pod \"ab1a2e42-7162-459d-817a-1913f54ec483\" (UID: \"ab1a2e42-7162-459d-817a-1913f54ec483\") " Oct 29 22:46:26.419201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419177 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.419329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419317 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.419437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419424 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzn9f\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-kube-api-access-fzn9f\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.419539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419526 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8z9g\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-kube-api-access-z8z9g\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.419625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419614 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.419710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419700 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.419798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419787 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147828d4-7a3e-4f50-964f-7132b042aec2-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.419898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419884 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.419994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.419982 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147828d4-7a3e-4f50-964f-7132b042aec2-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.420078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.420067 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147828d4-7a3e-4f50-964f-7132b042aec2-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.420156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.420147 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f1dce89-199e-4206-b6c6-d3fb049f64e8-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.420564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.420525 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-credential-socket" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.420898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.420867 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1a2e42-7162-459d-817a-1913f54ec483-istiod-ca-cert" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:26.421478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.421435 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-certs" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.421676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.421640 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-data" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.429249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.429213 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-socket" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.430263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.430207 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-envoy" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:26.430263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.430222 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab1a2e42-7162-459d-817a-1913f54ec483-istio-podinfo" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:26.430937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.430890 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-kube-api-access-k76lj" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "kube-api-access-k76lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.432503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.432466 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-istio-token" pod "ab1a2e42-7162-459d-817a-1913f54ec483" (UID: "ab1a2e42-7162-459d-817a-1913f54ec483"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:26.521915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.521876 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab1a2e42-7162-459d-817a-1913f54ec483-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.521915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.521916 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76lj\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-kube-api-access-k76lj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.522158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.521934 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab1a2e42-7162-459d-817a-1913f54ec483-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.522158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.521948 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab1a2e42-7162-459d-817a-1913f54ec483-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.522158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.521961 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.522158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.521976 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.522158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.521989 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.522158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.522004 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.522158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.522017 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab1a2e42-7162-459d-817a-1913f54ec483-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:26.570197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.570154 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp"] Oct 29 22:46:26.576838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.576807 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jztnp"] Oct 29 22:46:26.825671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:26.825419 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:27.110159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.110077 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" Oct 29 22:46:27.110159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.110084 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p" event={"ID":"147828d4-7a3e-4f50-964f-7132b042aec2","Type":"ContainerDied","Data":"b763bdaa56b746d3ad2ab6d856fdadaaffdd27c693b79e7526bc89c52ef78609"} Oct 29 22:46:27.110159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.110157 5610 scope.go:122] "RemoveContainer" containerID="0fe7244e3e5912626afd9048cf31cc54ca188269988f5b3842b9146a477bab36" Oct 29 22:46:27.111862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.111843 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" Oct 29 22:46:27.112107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.111852 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf" event={"ID":"ab1a2e42-7162-459d-817a-1913f54ec483","Type":"ContainerDied","Data":"91e4bb544d5891e74673088f5d19e896877773a948c5bb8f387a87d7f19e61d4"} Oct 29 22:46:27.114896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.114864 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" event={"ID":"9f1dce89-199e-4206-b6c6-d3fb049f64e8","Type":"ContainerDied","Data":"d604c217e2636695532fd092ec4d75459bd16be6d60ed646f96fca05c53beb31"} Oct 29 22:46:27.114896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.114888 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9" Oct 29 22:46:27.124543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.124519 5610 scope.go:122] "RemoveContainer" containerID="5b8c5a52249f29a93074001dc386a26f7b6f33d672ddc10369f42558f43b74a1" Oct 29 22:46:27.135446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.135419 5610 scope.go:122] "RemoveContainer" containerID="048cb6c6e3e403f43aa3d061a42a5cc441941ea537835e2e478db49d86813d0a" Oct 29 22:46:27.164274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.164240 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p"] Oct 29 22:46:27.176976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.176944 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-jtd6p"] Oct 29 22:46:27.203942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.203906 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf"] Oct 29 22:46:27.207679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.207654 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-hgnvf"] Oct 29 22:46:27.255087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.255057 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9"] Oct 29 22:46:27.258821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.258793 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63db1a-openshift-default-85c7894bbf-bjvk9"] Oct 29 22:46:27.642331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.642298 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" path="/var/lib/kubelet/pods/147828d4-7a3e-4f50-964f-7132b042aec2/volumes" Oct 29 22:46:27.643076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.643056 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" path="/var/lib/kubelet/pods/9f1dce89-199e-4206-b6c6-d3fb049f64e8/volumes" Oct 29 22:46:27.643745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.643725 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" path="/var/lib/kubelet/pods/9f653887-6403-42ab-96ee-dfc750ab4be3/volumes" Oct 29 22:46:27.644901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.644877 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" path="/var/lib/kubelet/pods/ab1a2e42-7162-459d-817a-1913f54ec483/volumes" Oct 29 22:46:27.748506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.748468 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:27.908744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:27.908629 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:28.850492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:28.837379 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:30.420225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:30.420187 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:31.863212 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.863149 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd"] Oct 29 22:46:31.864000 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.863982 5610 cpu_manager.go:414] "containerMap: removing container" podUID="171987f3-1f45-4938-a66f-702a5b025b7b" containerName="payload-pre-processing" Oct 29 22:46:31.864074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864000 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171987f3-1f45-4938-a66f-702a5b025b7b" containerName="payload-pre-processing" Oct 29 22:46:31.864074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864018 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" Oct 29 22:46:31.864074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864024 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" Oct 29 22:46:31.864074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864039 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" Oct 29 22:46:31.864074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864044 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864087 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864092 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864100 5610 cpu_manager.go:414] "containerMap: removing container" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864105 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864222 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f653887-6403-42ab-96ee-dfc750ab4be3" containerName="istio-proxy" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864229 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="147828d4-7a3e-4f50-964f-7132b042aec2" containerName="istio-proxy" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864236 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="171987f3-1f45-4938-a66f-702a5b025b7b" containerName="payload-pre-processing" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864242 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1a2e42-7162-459d-817a-1913f54ec483" containerName="istio-proxy" Oct 29 22:46:31.864325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.864251 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1dce89-199e-4206-b6c6-d3fb049f64e8" containerName="istio-proxy" Oct 29 22:46:31.867002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.866971 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:31.892664 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.892636 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-437981-serving-cert\"" Oct 29 22:46:31.913453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.899173 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd"] Oct 29 22:46:31.933643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.933614 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:31.933643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.933655 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/4474e487-1772-417f-82d8-c52932f1b21a-kube-api-access-h8lqg\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:31.933892 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.933722 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474e487-1772-417f-82d8-c52932f1b21a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:31.933892 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.933829 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:31.994788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:31.994752 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:32.035847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.035813 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.036057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.036038 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.036128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.036082 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/4474e487-1772-417f-82d8-c52932f1b21a-kube-api-access-h8lqg\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.036193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.036137 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474e487-1772-417f-82d8-c52932f1b21a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.036857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.036827 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474e487-1772-417f-82d8-c52932f1b21a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.038467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.038440 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.038563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.038460 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.055439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.055410 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/4474e487-1772-417f-82d8-c52932f1b21a-kube-api-access-h8lqg\") pod \"maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.148904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.148766 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:32.183049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.182701 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:32.537956 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:46:32.537921 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4474e487_1772_417f_82d8_c52932f1b21a.slice/crio-af5fb5f76fe7eec1ad9e91d9d33498d01ff4f7fc2f1d546d7dcf321806bcafc9 WatchSource:0}: Error finding container af5fb5f76fe7eec1ad9e91d9d33498d01ff4f7fc2f1d546d7dcf321806bcafc9: Status 404 returned error can't find the container with id af5fb5f76fe7eec1ad9e91d9d33498d01ff4f7fc2f1d546d7dcf321806bcafc9 Oct 29 22:46:32.539145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:32.539116 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd"] Oct 29 22:46:33.278224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:33.278188 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:46:33.367485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:33.363011 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" event={"ID":"4474e487-1772-417f-82d8-c52932f1b21a","Type":"ContainerStarted","Data":"bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0"} Oct 29 22:46:33.367485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:33.363073 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" event={"ID":"4474e487-1772-417f-82d8-c52932f1b21a","Type":"ContainerStarted","Data":"af5fb5f76fe7eec1ad9e91d9d33498d01ff4f7fc2f1d546d7dcf321806bcafc9"} Oct 29 22:46:33.367485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:33.364334 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:33.411202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:33.410197 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" podStartSLOduration=2.053714317 podStartE2EDuration="2.410179885s" totalImagesPullingTime="356.465568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:33.407822199 +0000 UTC m=+1855.889139692" watchObservedRunningTime="2026-10-29 22:46:33.410179885 +0000 UTC m=+1855.891497381" Oct 29 22:46:33.929541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:33.929506 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:35.565342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:35.550437 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:37.041818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:37.039422 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:38.635352 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:38.635311 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:38.780018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:38.779984 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:40.281117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:40.281074 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:40.384297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:40.384269 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 22:46:42.456423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:42.456300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:43.819842 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:46:43.819765 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:46:44.389895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:44.389821 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:46.079472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:46.077731 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:47.856870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:47.856834 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:48.011913 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:48.011878 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:48.444148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:48.444091 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:49.650424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:49.647040 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:50.563766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.563730 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms"] Oct 29 22:46:50.564032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.564006 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" containerID="cri-o://d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811" gracePeriod=30 Oct 29 22:46:50.849421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.849316 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs"] Oct 29 22:46:50.856474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.856442 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.892328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.892289 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs"] Oct 29 22:46:50.940540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.940495 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.940752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.940577 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.940752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.940710 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.940918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.940819 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.940918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.940883 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.941064 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.941035 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.941127 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.941102 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.941227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.941211 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:50.941278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:50.941257 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8jnv\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-kube-api-access-w8jnv\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.042887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.042849 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.042905 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.042939 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043017 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043057 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043101 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043145 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043197 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043237 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8jnv\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-kube-api-access-w8jnv\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043273 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043508 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043579 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043645 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.043839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.043815 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.045588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.045560 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.045719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.045701 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.063886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.063852 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8jnv\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-kube-api-access-w8jnv\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.064061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.064035 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.096300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.096261 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g"] Oct 29 22:46:51.104383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.103759 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.125764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.124728 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g"] Oct 29 22:46:51.125764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.125542 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8ebc60-serving-cert\"" Oct 29 22:46:51.144917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.144869 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6gqb\" (UniqueName: \"kubernetes.io/projected/f10315ab-7866-471f-9f67-bece06d8af03-kube-api-access-q6gqb\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.145106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.145079 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-metrics-certs\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.145195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.145126 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-maas-api-tls\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.145263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.145228 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10315ab-7866-471f-9f67-bece06d8af03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.198744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.198675 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:51.236653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.236586 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:51.247528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.247496 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10315ab-7866-471f-9f67-bece06d8af03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.247802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.247776 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6gqb\" (UniqueName: \"kubernetes.io/projected/f10315ab-7866-471f-9f67-bece06d8af03-kube-api-access-q6gqb\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.247938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.247922 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-metrics-certs\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.248003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.247958 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-maas-api-tls\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.248296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.248272 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10315ab-7866-471f-9f67-bece06d8af03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.250529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.250503 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-metrics-certs\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.250622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.250512 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-maas-api-tls\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.271749 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.271723 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6gqb\" (UniqueName: \"kubernetes.io/projected/f10315ab-7866-471f-9f67-bece06d8af03-kube-api-access-q6gqb\") pod \"maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.448254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.443094 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:51.448254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.444800 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:51.451843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.451808 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm"] Oct 29 22:46:51.452097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.452069 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" containerID="cri-o://bd83f1aecd1335833096de025140e2d416598611f673d4adbf2893bcbc1aa605" gracePeriod=30 Oct 29 22:46:51.764561 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.764203 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs"] Oct 29 22:46:51.765429 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:46:51.765362 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dbba23_277e_46f6_9050_7ceb0682b2b9.slice/crio-9fbc205f2f1a37a062cd2d62d5ffdf75c5b02a9f82c3749267466cfed53c910e WatchSource:0}: Error finding container 9fbc205f2f1a37a062cd2d62d5ffdf75c5b02a9f82c3749267466cfed53c910e: Status 404 returned error can't find the container with id 9fbc205f2f1a37a062cd2d62d5ffdf75c5b02a9f82c3749267466cfed53c910e Oct 29 22:46:51.769183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.769141 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:51.769296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.769231 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:51.769296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.769280 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:46:51.874686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:51.874655 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g"] Oct 29 22:46:51.877312 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:46:51.877283 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10315ab_7866_471f_9f67_bece06d8af03.slice/crio-ae4a2f1c7e5687f4b8a9f465bdf53cb901b1e66784f81ac4aaf1fd6307acd99a WatchSource:0}: Error finding container ae4a2f1c7e5687f4b8a9f465bdf53cb901b1e66784f81ac4aaf1fd6307acd99a: Status 404 returned error can't find the container with id ae4a2f1c7e5687f4b8a9f465bdf53cb901b1e66784f81ac4aaf1fd6307acd99a Oct 29 22:46:52.885241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:52.867835 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" event={"ID":"f10315ab-7866-471f-9f67-bece06d8af03","Type":"ContainerStarted","Data":"686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0"} Oct 29 22:46:52.885241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:52.867925 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" event={"ID":"f10315ab-7866-471f-9f67-bece06d8af03","Type":"ContainerStarted","Data":"ae4a2f1c7e5687f4b8a9f465bdf53cb901b1e66784f81ac4aaf1fd6307acd99a"} Oct 29 22:46:52.885241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:52.883340 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:46:52.904682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:52.904630 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" event={"ID":"b3dbba23-277e-46f6-9050-7ceb0682b2b9","Type":"ContainerStarted","Data":"060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649"} Oct 29 22:46:52.904892 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:52.904710 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" event={"ID":"b3dbba23-277e-46f6-9050-7ceb0682b2b9","Type":"ContainerStarted","Data":"9fbc205f2f1a37a062cd2d62d5ffdf75c5b02a9f82c3749267466cfed53c910e"} Oct 29 22:46:52.946590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:52.946532 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" podStartSLOduration=1.528773153 podStartE2EDuration="1.946516094s" totalImagesPullingTime="417.742941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:52.942853207 +0000 UTC m=+1875.424170701" watchObservedRunningTime="2026-10-29 22:46:52.946516094 +0000 UTC m=+1875.427833585" Oct 29 22:46:52.997692 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:52.997632 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podStartSLOduration=2.997613861 podStartE2EDuration="2.997613861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:52.993536292 +0000 UTC m=+1875.474853786" watchObservedRunningTime="2026-10-29 22:46:52.997613861 +0000 UTC m=+1875.478931355" Oct 29 22:46:53.200593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:53.200562 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:53.225418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:53.216546 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:46:53.225418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:53.216632 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:46:53.258230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:53.258189 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:54.201906 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:54.201872 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:46:54.202323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:54.201936 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:46:54.454682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:54.454592 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:54.607290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:54.607065 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:54.724600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:54.724510 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn"] Oct 29 22:46:54.724803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:54.724768 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" containerID="cri-o://ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85" gracePeriod=30 Oct 29 22:46:54.809418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:54.800776 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:55.139498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:55.132617 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:55.298415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:55.287862 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:46:55.298415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:55.287928 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:46:55.411124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:55.410999 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:55.581416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:55.573117 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:55.933025 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:55.933002 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:46:56.047814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.047717 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-kube-api-access-z5h76\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-kube-api-access-z5h76\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.047990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.047835 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-certs\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.047990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.047870 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-podinfo\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.047990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.047973 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-envoy\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.048178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.048094 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istiod-ca-cert\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.048178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.048134 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-data\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.048259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.048184 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-token\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-token\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.048259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.048207 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-certs" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.048259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.048229 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-credential-socket\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.048440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.048386 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-socket\") pod \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\" (UID: \"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93\") " Oct 29 22:46:56.048623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.048601 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-data" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.049137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049117 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.049137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049138 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.049345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049296 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-socket" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.050496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049640 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-credential-socket" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.050496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049651 5610 generic.go:356] "Generic (PLEG): container finished" podID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerID="d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811" exitCode=0 Oct 29 22:46:56.050496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049697 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" Oct 29 22:46:56.050496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049761 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" event={"ID":"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93","Type":"ContainerDied","Data":"d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811"} Oct 29 22:46:56.050496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049819 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms" event={"ID":"8e71b7e3-f1c1-45e2-a76a-1d70917e6b93","Type":"ContainerDied","Data":"60cce4f06833074a3e03ef5bd08f124adc1e6cd76647a0448156f61179da27e7"} Oct 29 22:46:56.050496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.049854 5610 scope.go:122] "RemoveContainer" containerID="d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811" Oct 29 22:46:56.050496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.050468 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istiod-ca-cert" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:56.050859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.050642 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-podinfo" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:56.050859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.050822 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-kube-api-access-z5h76" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "kube-api-access-z5h76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:56.052297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.052273 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-envoy" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:56.052827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.052805 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-token" pod "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" (UID: "8e71b7e3-f1c1-45e2-a76a-1d70917e6b93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:56.111276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.110907 5610 scope.go:122] "RemoveContainer" containerID="d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811" Oct 29 22:46:56.111429 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:46:56.111336 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811\": container with ID starting with d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811 not found: ID does not exist" containerID="d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811" Oct 29 22:46:56.111429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.111407 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811"} err="failed to get container status \"d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811\": rpc error: code = NotFound desc = could not find container \"d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811\": container with ID starting with d0c1bbddd62a1ed47c018cb65b7add52ab5999379dda94dd87e2d659fa2f0811 not found: ID does not exist" Oct 29 22:46:56.174416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.155988 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.174416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.156034 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.174416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.156051 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.174416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.156066 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.174416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.156081 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5h76\" (UniqueName: \"kubernetes.io/projected/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-kube-api-access-z5h76\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.174416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.156094 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.174416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.156107 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:56.217434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.213703 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:46:56.217434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.213772 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:46:56.407543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.407495 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms"] Oct 29 22:46:56.411502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:56.411474 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-797ms"] Oct 29 22:46:57.106655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.104889 5610 generic.go:356] "Generic (PLEG): container finished" podID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerID="bd83f1aecd1335833096de025140e2d416598611f673d4adbf2893bcbc1aa605" exitCode=0 Oct 29 22:46:57.106655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.105007 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" event={"ID":"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75","Type":"ContainerDied","Data":"bd83f1aecd1335833096de025140e2d416598611f673d4adbf2893bcbc1aa605"} Oct 29 22:46:57.146525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.146496 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:46:57.203666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.200608 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:46:57.203666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.200682 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.220404 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-envoy\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.220461 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-kube-api-access-fsmkm\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-kube-api-access-fsmkm\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.220529 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-data\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.220637 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-podinfo\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.220670 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-token\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-token\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.220786 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istiod-ca-cert\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.220818 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-certs\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.220949 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-socket\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.221381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.221079 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-credential-socket\") pod \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\" (UID: \"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75\") " Oct 29 22:46:57.222860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.222830 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-data" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.223214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.223194 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-certs" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.223827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.223735 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istiod-ca-cert" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:46:57.227588 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.227123 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-envoy" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.228366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.228329 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-kube-api-access-fsmkm" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "kube-api-access-fsmkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:57.228499 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.228485 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-podinfo" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:46:57.228897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.228873 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-token" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:46:57.233462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.229498 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-credential-socket" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.233462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.229560 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-socket" pod "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" (UID: "4c17cc90-a9be-43bd-b7e9-2b556aa4fa75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:46:57.293146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.292997 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:57.323629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323594 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.323629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323628 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.323896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323642 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.323896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323659 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.323896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323677 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.323896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323692 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsmkm\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-kube-api-access-fsmkm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.323896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323705 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.323896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323719 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.323896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.323731 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:46:57.639335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:57.639254 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" path="/var/lib/kubelet/pods/8e71b7e3-f1c1-45e2-a76a-1d70917e6b93/volumes" Oct 29 22:46:58.116931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:58.116895 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" Oct 29 22:46:58.117148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:58.116943 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm" event={"ID":"4c17cc90-a9be-43bd-b7e9-2b556aa4fa75","Type":"ContainerDied","Data":"aed7c9ad80c0295b60570733219bf4ea45cf2e5a6b55aad5380145fcf8dc1090"} Oct 29 22:46:58.117148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:58.117013 5610 scope.go:122] "RemoveContainer" containerID="bd83f1aecd1335833096de025140e2d416598611f673d4adbf2893bcbc1aa605" Oct 29 22:46:58.195904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:58.195866 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm"] Oct 29 22:46:58.199867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:58.199833 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 29 22:46:58.200033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:58.199873 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 29 22:46:58.202684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:58.202656 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-vrjzm"] Oct 29 22:46:59.147321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:59.147275 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:46:59.205090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:59.205062 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:59.205618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:59.205604 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:59.206281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:59.206260 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:46:59.639099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:46:59.639064 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" path="/var/lib/kubelet/pods/4c17cc90-a9be-43bd-b7e9-2b556aa4fa75/volumes" Oct 29 22:47:00.032492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.032468 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:47:00.098055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.098027 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:47:00.130858 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.130822 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9943a5e-db25-4438-bdf0-74979290b12e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9943a5e-db25-4438-bdf0-74979290b12e-istiod-ca-cert\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.130924 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-credential-socket\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.130954 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-kube-api-access-dg9pd\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-kube-api-access-dg9pd\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131006 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-istio-token\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-istio-token\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131038 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-certs\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131096 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-socket\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131264 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-credential-socket" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:00.131311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131306 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-envoy\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131439 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9943a5e-db25-4438-bdf0-74979290b12e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9943a5e-db25-4438-bdf0-74979290b12e-istio-podinfo\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131486 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9943a5e-db25-4438-bdf0-74979290b12e-istiod-ca-cert" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:47:00.131653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131518 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-data\") pod \"a9943a5e-db25-4438-bdf0-74979290b12e\" (UID: \"a9943a5e-db25-4438-bdf0-74979290b12e\") " Oct 29 22:47:00.131803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131783 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-socket" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:00.131989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.131970 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-data" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:00.132536 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.132510 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.132654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.132555 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9943a5e-db25-4438-bdf0-74979290b12e-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.132654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.132571 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.132654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.132588 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.133098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.133072 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-certs" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:00.134047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.134017 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-istio-token" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:00.134991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.134961 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-kube-api-access-dg9pd" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "kube-api-access-dg9pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:47:00.135098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.135082 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9943a5e-db25-4438-bdf0-74979290b12e-istio-podinfo" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:47:00.135179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.135155 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-envoy" pod "a9943a5e-db25-4438-bdf0-74979290b12e" (UID: "a9943a5e-db25-4438-bdf0-74979290b12e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:47:00.139446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.139419 5610 generic.go:356] "Generic (PLEG): container finished" podID="a9943a5e-db25-4438-bdf0-74979290b12e" containerID="ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85" exitCode=0 Oct 29 22:47:00.140528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.140510 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" Oct 29 22:47:00.143681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.143524 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" event={"ID":"a9943a5e-db25-4438-bdf0-74979290b12e","Type":"ContainerDied","Data":"ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85"} Oct 29 22:47:00.143681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.143593 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn" event={"ID":"a9943a5e-db25-4438-bdf0-74979290b12e","Type":"ContainerDied","Data":"0e02333b94e416f31cd4ecaf7649f9e352ab6c507d86d57de3b681b1ebf05bdf"} Oct 29 22:47:00.143681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.143626 5610 scope.go:122] "RemoveContainer" containerID="ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85" Oct 29 22:47:00.190989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.190950 5610 scope.go:122] "RemoveContainer" containerID="772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec" Oct 29 22:47:00.208804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.208778 5610 scope.go:122] "RemoveContainer" containerID="ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85" Oct 29 22:47:00.209145 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:47:00.209122 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85\": container with ID starting with ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85 not found: ID does not exist" containerID="ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85" Oct 29 22:47:00.209247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.209159 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85"} err="failed to get container status \"ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85\": rpc error: code = NotFound desc = could not find container \"ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85\": container with ID starting with ea423842d4c2ebb45da9f2cfda25018b4809658b521960a17900aad8cb3dcf85 not found: ID does not exist" Oct 29 22:47:00.209247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.209186 5610 scope.go:122] "RemoveContainer" containerID="772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec" Oct 29 22:47:00.209594 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:47:00.209571 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec\": container with ID starting with 772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec not found: ID does not exist" containerID="772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec" Oct 29 22:47:00.209685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.209602 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec"} err="failed to get container status \"772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec\": rpc error: code = NotFound desc = could not find container \"772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec\": container with ID starting with 772913eda6277206e62df3275196376e8f7c5a009e7023ac2e2ffc348bf070ec not found: ID does not exist" Oct 29 22:47:00.221660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.221635 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn"] Oct 29 22:47:00.225875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.225846 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e224cd-openshift-default-67657fc945-d42qn"] Oct 29 22:47:00.234336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.234308 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg9pd\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-kube-api-access-dg9pd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.234443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.234339 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9943a5e-db25-4438-bdf0-74979290b12e-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.234443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.234358 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.234443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.234371 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9943a5e-db25-4438-bdf0-74979290b12e-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.234443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.234384 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9943a5e-db25-4438-bdf0-74979290b12e-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:47:00.725417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:00.725356 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:01.643514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:01.643476 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" path="/var/lib/kubelet/pods/a9943a5e-db25-4438-bdf0-74979290b12e/volumes" Oct 29 22:47:02.265323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:02.265271 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:03.860158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:03.860121 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:05.352665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.352616 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:05.878622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.878585 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4"] Oct 29 22:47:05.879420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879404 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" Oct 29 22:47:05.879476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879421 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" Oct 29 22:47:05.879476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879453 5610 cpu_manager.go:414] "containerMap: removing container" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" Oct 29 22:47:05.879476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879458 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" Oct 29 22:47:05.879585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879483 5610 cpu_manager.go:414] "containerMap: removing container" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" Oct 29 22:47:05.879585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879488 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" Oct 29 22:47:05.879585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879511 5610 cpu_manager.go:414] "containerMap: removing container" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" Oct 29 22:47:05.879585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879516 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" Oct 29 22:47:05.879707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879623 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" Oct 29 22:47:05.879707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879634 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c17cc90-a9be-43bd-b7e9-2b556aa4fa75" containerName="istio-proxy" Oct 29 22:47:05.879707 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.879641 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e71b7e3-f1c1-45e2-a76a-1d70917e6b93" containerName="istio-proxy" Oct 29 22:47:05.883750 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.883731 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.922446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.919849 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4"] Oct 29 22:47:05.953945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.953910 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.953945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.953946 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.954185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.954057 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.954185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.954154 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.954185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.954181 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdz2\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-kube-api-access-gpdz2\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.954306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.954265 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.954362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.954306 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.954362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.954342 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:05.954497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:05.954479 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.056324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.056282 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.056543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.056357 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.056543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.056422 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.056543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.056495 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.056787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.056618 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.056787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.056649 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.056787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.056703 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.057005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.056978 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.057417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.057367 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdz2\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-kube-api-access-gpdz2\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.057603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.057575 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.057696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.057258 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istiod-ca-cert\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.057914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.057887 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-credential-socket\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.057995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.057906 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-certs\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.058111 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.058090 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-data\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.059172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.059151 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-envoy\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.059364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.059343 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-podinfo\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.076227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.076194 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-token\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.076363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.076262 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdz2\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-kube-api-access-gpdz2\") pod \"e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.202489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.202457 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:06.522807 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:47:06.522774 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788d8d73_dc49_4bc5_aac2_1a8cbf4e5d10.slice/crio-d79a4eef3f3a5181d5c87cc44667928ad1f24d1f892f92adc08ab717a5f11b3f WatchSource:0}: Error finding container d79a4eef3f3a5181d5c87cc44667928ad1f24d1f892f92adc08ab717a5f11b3f: Status 404 returned error can't find the container with id d79a4eef3f3a5181d5c87cc44667928ad1f24d1f892f92adc08ab717a5f11b3f Oct 29 22:47:06.523807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.523780 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4"] Oct 29 22:47:06.525987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.525954 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:06.526081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.526024 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:06.526081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.526058 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:06.941961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:06.941858 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:07.204542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:07.204495 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" event={"ID":"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10","Type":"ContainerStarted","Data":"2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67"} Oct 29 22:47:07.204732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:07.204562 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" event={"ID":"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10","Type":"ContainerStarted","Data":"d79a4eef3f3a5181d5c87cc44667928ad1f24d1f892f92adc08ab717a5f11b3f"} Oct 29 22:47:07.248698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:07.248643 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" podStartSLOduration=2.24862754 podStartE2EDuration="2.24862754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:07.247135393 +0000 UTC m=+1889.728452901" watchObservedRunningTime="2026-10-29 22:47:07.24862754 +0000 UTC m=+1889.729945032" Oct 29 22:47:08.202696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:08.202658 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:08.203942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:08.203913 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:47:08.204057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:08.203988 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 22:47:08.622164 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:08.622084 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:08.766429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:08.764965 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:08.908598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:08.908504 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:09.203365 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:09.203328 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:47:09.203913 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:09.203388 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 22:47:10.205232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:10.205200 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:47:10.205713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:10.205688 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 22:47:10.672115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:10.672020 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:11.203144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:11.203109 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:47:11.203431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:11.203373 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 22:47:12.218532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:12.217742 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 29 22:47:12.218532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:12.217886 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 29 22:47:12.904491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:12.904451 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:13.209702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:13.209671 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:13.210097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:13.210069 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:13.210809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:13.210785 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:47:13.717746 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:47:13.717704 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:47:13.908531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:13.908493 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:14.797945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:14.797910 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:16.333034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:16.332974 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:17.943412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:17.943346 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:19.627386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:19.627326 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:20.614093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:20.614055 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:21.095834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.095789 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:21.757748 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.757701 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw"] Oct 29 22:47:21.760524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.759603 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9943a5e-db25-4438-bdf0-74979290b12e" containerName="istio-proxy" Oct 29 22:47:21.783246 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.781786 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:21.783246 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.782976 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.812461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.812427 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-dbb507-openshift-default-dockercfg-54prz\"" Oct 29 22:47:21.831594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.831507 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw"] Oct 29 22:47:21.871652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.871614 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.871857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.871725 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-credential-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.871857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.871753 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-certs\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.871857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.871780 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-podinfo\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.872033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.871880 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-token\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.872033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.871929 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-envoy\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.872033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.871998 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77qp7\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-kube-api-access-77qp7\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.872215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.872040 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-data\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:21.872215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.872109 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istiod-ca-cert\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.002429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.998426 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qp7\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-kube-api-access-77qp7\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.002429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.998501 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-data\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.002429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.998572 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istiod-ca-cert\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.002429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.998742 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:22.002429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.998772 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.002429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:21.999203 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-data\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.002429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.000200 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istiod-ca-cert\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.002429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.000564 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.026459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.022537 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-credential-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.026459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.022581 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-certs\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.026459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.022608 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-podinfo\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.026459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.022675 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-token\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.026459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.022727 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-envoy\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.026459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.023167 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-certs\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.045244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.033500 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-credential-socket\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.045244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.043573 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-envoy\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.046529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.046344 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77qp7\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-kube-api-access-77qp7\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.046888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.046830 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-podinfo\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.049199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.049132 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-token\") pod \"e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.152912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.152859 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:22.228630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.213821 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:22.551269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.551239 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw"] Oct 29 22:47:22.551910 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:47:22.551888 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2d8c3e_e6b1_47a5_a26d_1d923d92a685.slice/crio-c2bedddc9f8cb2122cb614176ea0ebb17043ef941312ea7f7a6fe3480c5c70cc WatchSource:0}: Error finding container c2bedddc9f8cb2122cb614176ea0ebb17043ef941312ea7f7a6fe3480c5c70cc: Status 404 returned error can't find the container with id c2bedddc9f8cb2122cb614176ea0ebb17043ef941312ea7f7a6fe3480c5c70cc Oct 29 22:47:22.554840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.554803 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:22.554943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.554875 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:22.554943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.554904 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:22.575685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:22.575644 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" event={"ID":"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685","Type":"ContainerStarted","Data":"c2bedddc9f8cb2122cb614176ea0ebb17043ef941312ea7f7a6fe3480c5c70cc"} Oct 29 22:47:23.602427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:23.599912 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:23.603204 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:23.603166 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" event={"ID":"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685","Type":"ContainerStarted","Data":"054f76c1d32023c5b85579c7f24d44490023beec38f08a788c78e17a0bca6279"} Oct 29 22:47:23.651344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:23.651286 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podStartSLOduration=2.651267554 podStartE2EDuration="2.651267554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:23.649776966 +0000 UTC m=+1906.131094460" watchObservedRunningTime="2026-10-29 22:47:23.651267554 +0000 UTC m=+1906.132585048" Oct 29 22:47:24.153865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:24.153829 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:24.160693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:24.160666 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:24.641478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:24.640441 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:24.666471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:24.642847 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:47:26.329500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:26.325937 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:26.593462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:26.591093 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:28.750413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:28.749024 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:29.233044 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.203851 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:29.489515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.476977 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:29.516148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.516115 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf"] Oct 29 22:47:29.530441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.530409 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:29.532567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.532541 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf"] Oct 29 22:47:29.542942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.542919 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmms7\" (UniqueName: \"kubernetes.io/projected/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-kube-api-access-cmms7\") pod \"payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf\" (UID: \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:29.543289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.543271 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf\" (UID: \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:29.545732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.545710 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-dbb507\"" Oct 29 22:47:29.556950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.556923 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-dbb507-dockercfg-vwcft\"" Oct 29 22:47:29.644450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.644384 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf\" (UID: \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:29.644625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.644480 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmms7\" (UniqueName: \"kubernetes.io/projected/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-kube-api-access-cmms7\") pod \"payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf\" (UID: \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:29.645248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.645223 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf\" (UID: \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:29.708417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.704593 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmms7\" (UniqueName: \"kubernetes.io/projected/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-kube-api-access-cmms7\") pod \"payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf\" (UID: \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:29.849177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:29.849098 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:30.332789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:30.327463 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf"] Oct 29 22:47:30.858052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:30.858018 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" event={"ID":"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4","Type":"ContainerStarted","Data":"7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5"} Oct 29 22:47:30.858476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:30.858070 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" event={"ID":"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4","Type":"ContainerStarted","Data":"4fe7a9f6f9a3c59cb187e6a637fef580a02e4c3067b62f7718f2dc5417c5c806"} Oct 29 22:47:30.858798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:30.858780 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:30.913420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:30.910618 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" podStartSLOduration=2.910598737 podStartE2EDuration="2.910598737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:30.909860349 +0000 UTC m=+1913.391177846" watchObservedRunningTime="2026-10-29 22:47:30.910598737 +0000 UTC m=+1913.391916231" Oct 29 22:47:31.565430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:31.561979 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:33.441691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.441653 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z"] Oct 29 22:47:33.450535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.450496 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z"] Oct 29 22:47:33.450712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.450613 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.490263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.486647 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-379a6d\"/\"kube-root-ca.crt\"" Oct 29 22:47:33.490263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.486769 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-379a6d\"/\"openshift-service-ca.crt\"" Oct 29 22:47:33.648432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.581118 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-379a6d\"/\"test-model-379a6d-kserve-self-signed-certs\"" Oct 29 22:47:33.740520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.609009 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-379a6d\"/\"default-dockercfg-rb5lj\"" Oct 29 22:47:33.773803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.643108 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8cb\" (UniqueName: \"kubernetes.io/projected/ecb5ed71-07f9-4334-aab8-cef607b23d5d-kube-api-access-8h8cb\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.774100 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.774071 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tmp-dir\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.774607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.774579 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-home\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.793498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.784516 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-dshm\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.793498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.784805 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-model-cache\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.793498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.784933 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tls-certs\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.800419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.800270 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:33.897427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.894322 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-model-cache\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.897427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.894498 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tls-certs\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.897427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.894621 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8cb\" (UniqueName: \"kubernetes.io/projected/ecb5ed71-07f9-4334-aab8-cef607b23d5d-kube-api-access-8h8cb\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.897427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.894668 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tmp-dir\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.897427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.894745 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-home\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.897427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.894861 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-dshm\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.899578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.898248 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-model-cache\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.899578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.898722 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tmp-dir\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.899578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.899319 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-home\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.930531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.929706 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-dshm\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.945803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.945772 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tls-certs\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.946130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.946097 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8cb\" (UniqueName: \"kubernetes.io/projected/ecb5ed71-07f9-4334-aab8-cef607b23d5d-kube-api-access-8h8cb\") pod \"test-model-379a6d-kserve-df64688c-qfr6z\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:33.975427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:33.962051 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:34.084768 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:34.084723 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:34.406856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:34.406819 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z"] Oct 29 22:47:34.407909 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:47:34.407875 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb5ed71_07f9_4334_aab8_cef607b23d5d.slice/crio-ce06a48fa53a10c475aa51eded1c5d3b119ea8a09294c21b41072f111acd8ad1 WatchSource:0}: Error finding container ce06a48fa53a10c475aa51eded1c5d3b119ea8a09294c21b41072f111acd8ad1: Status 404 returned error can't find the container with id ce06a48fa53a10c475aa51eded1c5d3b119ea8a09294c21b41072f111acd8ad1 Oct 29 22:47:35.020471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:35.020431 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" event={"ID":"ecb5ed71-07f9-4334-aab8-cef607b23d5d","Type":"ContainerStarted","Data":"c7b3946f801b8efe2b0c15e13f9496312e22c976ac7b180aff88580316d67c4c"} Oct 29 22:47:35.020471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:35.020476 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" event={"ID":"ecb5ed71-07f9-4334-aab8-cef607b23d5d","Type":"ContainerStarted","Data":"ce06a48fa53a10c475aa51eded1c5d3b119ea8a09294c21b41072f111acd8ad1"} Oct 29 22:47:35.067931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:35.067864 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" podStartSLOduration=2.067843943 podStartE2EDuration="2.067843943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:47:35.059825954 +0000 UTC m=+1917.541143448" watchObservedRunningTime="2026-10-29 22:47:35.067843943 +0000 UTC m=+1917.549161434" Oct 29 22:47:35.085632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:35.085601 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:35.475416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:35.473220 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:37.174550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:37.174512 5610 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:47:37.175222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:37.174578 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:47:37.240600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:37.240563 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:38.776978 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:38.776931 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:38.934835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:38.934748 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:39.071921 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:39.071835 5610 generic.go:356] "Generic (PLEG): container finished" podID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerID="14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab" exitCode=0 Oct 29 22:47:39.072077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:39.071916 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" event={"ID":"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02","Type":"ContainerDied","Data":"14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab"} Oct 29 22:47:39.072553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:39.072531 5610 scope.go:122] "RemoveContainer" containerID="14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab" Oct 29 22:47:39.074736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:39.074705 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:39.074860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:39.074793 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:39.074860 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:39.074843 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:40.099662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.099626 5610 generic.go:356] "Generic (PLEG): container finished" podID="8ea5c830-31e9-498e-a774-9c3400e83178" containerID="6162de3e1c77f15f766b1d4af933e529ff9e8eca9a21d2b6d8cd6c4dd6b7afca" exitCode=0 Oct 29 22:47:40.100194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.099717 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" event={"ID":"8ea5c830-31e9-498e-a774-9c3400e83178","Type":"ContainerDied","Data":"6162de3e1c77f15f766b1d4af933e529ff9e8eca9a21d2b6d8cd6c4dd6b7afca"} Oct 29 22:47:40.100682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.100663 5610 scope.go:122] "RemoveContainer" containerID="6162de3e1c77f15f766b1d4af933e529ff9e8eca9a21d2b6d8cd6c4dd6b7afca" Oct 29 22:47:40.103755 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.103080 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:40.103755 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.103153 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:40.103755 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.103200 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:47:40.110598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.109801 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" event={"ID":"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02","Type":"ContainerStarted","Data":"c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b"} Oct 29 22:47:40.252602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.252573 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:47:40.252788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.252684 5610 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:47:40.252788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.252727 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:47:40.536418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:40.532324 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:41.119663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:41.119630 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" event={"ID":"8ea5c830-31e9-498e-a774-9c3400e83178","Type":"ContainerStarted","Data":"34f377e425dfb7b9c92ddbfeff28c0aebfd7180b4e3bb2cd43d2150e5d39132b"} Oct 29 22:47:41.252376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:41.252342 5610 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:47:41.252560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:41.252422 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:47:41.925414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:41.917056 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:47:41.925414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:41.917247 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:47:41.925414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:41.917307 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:47:42.293416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:42.281983 5610 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:47:42.293416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:42.282060 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:47:42.651470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:42.642694 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:42.781419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:42.780589 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:47:42.922472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:42.922013 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:47:42.922472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:42.922111 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:47:42.933135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:42.933098 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 22:47:43.252529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:43.252496 5610 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:47:43.252711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:43.252542 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:47:43.755521 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:47:43.755482 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:47:43.925422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:43.925296 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:47:43.925422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:43.925365 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:47:44.098531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:44.098439 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:44.136555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:44.136527 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:44.252866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:44.252835 5610 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:47:44.253035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:44.252884 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:47:44.357719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:44.357635 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:47:44.829175 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:44.829067 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:44.923418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:44.920017 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:47:44.923418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:44.920160 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:47:45.252963 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:45.252932 5610 patch_prober.go:29] interesting pod/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 29 22:47:45.253131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:45.252982 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 29 22:47:45.915848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:45.915804 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:47:45.916263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:45.915875 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:47:46.256677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:46.256645 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:47:46.257083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:46.257061 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:47:46.257776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:46.257754 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:47:46.936423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:46.935517 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:47:46.936423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:46.935584 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:47:46.936423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:46.936338 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:47.093258 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:47.093223 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:47.562513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:47.559929 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:47.916657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:47.916572 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 29 22:47:47.916657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:47.916634 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 29 22:47:48.920581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:48.920547 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:47:48.921753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:48.921731 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 22:47:49.111419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:49.109008 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:49.579422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:49.578710 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:50.988573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:50.988529 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:51.621295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:51.621259 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:52.962418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:52.961368 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:54.947160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:54.947118 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:56.631373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:56.631332 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:47:58.473165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:47:58.473126 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:00.346062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:00.346017 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:02.304417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:02.304235 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:03.223798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:03.223759 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:03.688313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:03.688211 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:04.202072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:04.202034 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:04.394309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:04.394269 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:06.254464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:06.248983 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:07.985430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:07.985299 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:08.211484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:08.211445 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:08.450535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:08.450481 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:09.882168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:09.882120 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:10.048756 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.048700 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:10.090209 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.090173 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8"] Oct 29 22:48:10.094434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.094411 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:10.108909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.108884 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-57d336\"" Oct 29 22:48:10.109048 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.108973 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-57d336-dockercfg-p5xxm\"" Oct 29 22:48:10.124515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.124480 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8"] Oct 29 22:48:10.132984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.132887 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88pd\" (UniqueName: \"kubernetes.io/projected/c25ff675-008c-46fa-808f-ea1bf5a30670-kube-api-access-z88pd\") pod \"payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8\" (UID: \"c25ff675-008c-46fa-808f-ea1bf5a30670\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:10.133328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.133297 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ff675-008c-46fa-808f-ea1bf5a30670-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8\" (UID: \"c25ff675-008c-46fa-808f-ea1bf5a30670\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:10.242417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.242290 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ff675-008c-46fa-808f-ea1bf5a30670-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8\" (UID: \"c25ff675-008c-46fa-808f-ea1bf5a30670\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:10.258440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.255695 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88pd\" (UniqueName: \"kubernetes.io/projected/c25ff675-008c-46fa-808f-ea1bf5a30670-kube-api-access-z88pd\") pod \"payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8\" (UID: \"c25ff675-008c-46fa-808f-ea1bf5a30670\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:10.258440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.257356 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ff675-008c-46fa-808f-ea1bf5a30670-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8\" (UID: \"c25ff675-008c-46fa-808f-ea1bf5a30670\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:10.592231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.592195 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw"] Oct 29 22:48:10.596961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.596927 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88pd\" (UniqueName: \"kubernetes.io/projected/c25ff675-008c-46fa-808f-ea1bf5a30670-kube-api-access-z88pd\") pod \"payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8\" (UID: \"c25ff675-008c-46fa-808f-ea1bf5a30670\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:10.601923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.601708 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw"] Oct 29 22:48:10.602158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.602140 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:10.705560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.705524 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwmv\" (UniqueName: \"kubernetes.io/projected/8191db7b-cb6a-4aae-b00c-c2232aa8f278-kube-api-access-bzwmv\") pod \"payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw\" (UID: \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\") " pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:10.705784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.705651 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8191db7b-cb6a-4aae-b00c-c2232aa8f278-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw\" (UID: \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\") " pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:10.709673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.709644 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:10.808045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.808000 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwmv\" (UniqueName: \"kubernetes.io/projected/8191db7b-cb6a-4aae-b00c-c2232aa8f278-kube-api-access-bzwmv\") pod \"payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw\" (UID: \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\") " pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:10.808298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.808087 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8191db7b-cb6a-4aae-b00c-c2232aa8f278-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw\" (UID: \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\") " pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:10.809293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.809259 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8191db7b-cb6a-4aae-b00c-c2232aa8f278-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw\" (UID: \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\") " pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:10.838074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.837937 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwmv\" (UniqueName: \"kubernetes.io/projected/8191db7b-cb6a-4aae-b00c-c2232aa8f278-kube-api-access-bzwmv\") pod \"payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw\" (UID: \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\") " pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:10.922195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:10.922108 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:11.037858 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.037827 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8"] Oct 29 22:48:11.042048 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:48:11.042022 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25ff675_008c_46fa_808f_ea1bf5a30670.slice/crio-03a71992b5d053ff2b1352caf545299feff86cfccc1d80d08d804554d86fe05d WatchSource:0}: Error finding container 03a71992b5d053ff2b1352caf545299feff86cfccc1d80d08d804554d86fe05d: Status 404 returned error can't find the container with id 03a71992b5d053ff2b1352caf545299feff86cfccc1d80d08d804554d86fe05d Oct 29 22:48:11.237639 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:48:11.237596 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8191db7b_cb6a_4aae_b00c_c2232aa8f278.slice/crio-8f56a3cadd7e3251a165021a5be8c5f00632f68e4ff76c2d18e245d2eb09099c WatchSource:0}: Error finding container 8f56a3cadd7e3251a165021a5be8c5f00632f68e4ff76c2d18e245d2eb09099c: Status 404 returned error can't find the container with id 8f56a3cadd7e3251a165021a5be8c5f00632f68e4ff76c2d18e245d2eb09099c Oct 29 22:48:11.237639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.237602 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw"] Oct 29 22:48:11.667762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.667657 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" event={"ID":"8191db7b-cb6a-4aae-b00c-c2232aa8f278","Type":"ContainerStarted","Data":"d24c998fd0621864d3f7f016481efff8b537dd70b13a0fe27152dafe2d61d902"} Oct 29 22:48:11.667762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.667721 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" event={"ID":"8191db7b-cb6a-4aae-b00c-c2232aa8f278","Type":"ContainerStarted","Data":"8f56a3cadd7e3251a165021a5be8c5f00632f68e4ff76c2d18e245d2eb09099c"} Oct 29 22:48:11.668000 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.667966 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:11.669684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.669656 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" event={"ID":"c25ff675-008c-46fa-808f-ea1bf5a30670","Type":"ContainerStarted","Data":"c766bf93ba3e6ca8c824e653da10efdd51997807e0ed6d1db76bb7f5926290f5"} Oct 29 22:48:11.669850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.669706 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:11.669850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.669726 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" event={"ID":"c25ff675-008c-46fa-808f-ea1bf5a30670","Type":"ContainerStarted","Data":"03a71992b5d053ff2b1352caf545299feff86cfccc1d80d08d804554d86fe05d"} Oct 29 22:48:11.707214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.707156 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" podStartSLOduration=1.70714093 podStartE2EDuration="1.70714093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:11.706215598 +0000 UTC m=+1954.187533093" watchObservedRunningTime="2026-10-29 22:48:11.70714093 +0000 UTC m=+1954.188458428" Oct 29 22:48:11.749032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.748973 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" podStartSLOduration=1.748958162 podStartE2EDuration="1.748958162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:11.748274439 +0000 UTC m=+1954.229591929" watchObservedRunningTime="2026-10-29 22:48:11.748958162 +0000 UTC m=+1954.230275654" Oct 29 22:48:11.811142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.811102 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:11.822540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:11.822510 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp"] Oct 29 22:48:12.035423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:12.034025 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:13.549488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:13.542960 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:13.713947 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:48:13.713904 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:48:15.024421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:15.020162 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:16.802417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:16.794506 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:16.938476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:16.938442 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:17.502630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:17.502599 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:18.667423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:18.662032 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:20.269415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:20.265549 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:21.465955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:21.465912 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd"] Oct 29 22:48:21.466360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:21.466117 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" containerID="cri-o://8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b" gracePeriod=30 Oct 29 22:48:21.838445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:21.838336 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:22.683473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:22.683441 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:48:22.683917 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:22.683510 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:48:23.480274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:23.480235 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:23.651375 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:23.651298 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:25.153801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:25.153762 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:25.617251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:25.617218 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:26.627509 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.627478 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8"] Oct 29 22:48:26.632337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.632313 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.645053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.645002 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8ebc60\"/\"openshift-service-ca.crt\"" Oct 29 22:48:26.662419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.661239 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8ebc60\"/\"default-dockercfg-dzqbk\"" Oct 29 22:48:26.662419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.661438 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8ebc60\"/\"test-model-8ebc60-kserve-self-signed-certs\"" Oct 29 22:48:26.667418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.664821 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8"] Oct 29 22:48:26.746851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.746743 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-tmp-dir\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.746851 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.746813 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn52c\" (UniqueName: \"kubernetes.io/projected/bc290c4e-3d45-4547-bedc-ef18c50947b6-kube-api-access-mn52c\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.747110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.746881 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-model-cache\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.747110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.746947 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-home\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.747110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.747026 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc290c4e-3d45-4547-bedc-ef18c50947b6-tls-certs\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.747110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.747044 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-dshm\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.747983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.747942 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8ebc60\"/\"kube-root-ca.crt\"" Oct 29 22:48:26.751183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.751147 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:26.829719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.829688 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:48:26.846151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.846112 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" podUID="0b1ada16-8a79-4fd4-9320-e38c6f539e97" containerName="main" containerID="cri-o://4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b" gracePeriod=45 Oct 29 22:48:26.848409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.848376 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-home\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.848516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.848483 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc290c4e-3d45-4547-bedc-ef18c50947b6-tls-certs\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.848516 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.848509 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-dshm\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.848699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.848679 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-tmp-dir\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.848896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.848877 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn52c\" (UniqueName: \"kubernetes.io/projected/bc290c4e-3d45-4547-bedc-ef18c50947b6-kube-api-access-mn52c\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.849320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.848801 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-home\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.849320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.849099 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-model-cache\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.849320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.849111 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-tmp-dir\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.849524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.849336 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-model-cache\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.851525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.851490 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc290c4e-3d45-4547-bedc-ef18c50947b6-tls-certs\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.852322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.852303 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-dshm\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.864891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.864675 5610 generic.go:356] "Generic (PLEG): container finished" podID="7635edde-2274-444f-a076-b39842f07f12" containerID="8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b" exitCode=0 Oct 29 22:48:26.864891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.864709 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" event={"ID":"7635edde-2274-444f-a076-b39842f07f12","Type":"ContainerDied","Data":"8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b"} Oct 29 22:48:26.864891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.864759 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" Oct 29 22:48:26.864891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.864786 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd" event={"ID":"7635edde-2274-444f-a076-b39842f07f12","Type":"ContainerDied","Data":"f10bf86e1581df5833851c5855c759611c0cbe75ccd3e67e09a7c112b1e2896d"} Oct 29 22:48:26.864891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.864822 5610 scope.go:122] "RemoveContainer" containerID="8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b" Oct 29 22:48:26.866663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.866641 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn52c\" (UniqueName: \"kubernetes.io/projected/bc290c4e-3d45-4547-bedc-ef18c50947b6-kube-api-access-mn52c\") pod \"test-model-8ebc60-kserve-6f745f57ff-dvvp8\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.910537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.899861 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951063 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7635edde-2274-444f-a076-b39842f07f12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7635edde-2274-444f-a076-b39842f07f12-istio-podinfo\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951265 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-data\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951351 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-kube-api-access-v6pj6\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-kube-api-access-v6pj6\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951437 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-certs\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951468 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-istio-token\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-istio-token\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951518 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-socket\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951566 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-envoy\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951627 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7635edde-2274-444f-a076-b39842f07f12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7635edde-2274-444f-a076-b39842f07f12-istiod-ca-cert\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.956432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.951744 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-credential-socket\") pod \"7635edde-2274-444f-a076-b39842f07f12\" (UID: \"7635edde-2274-444f-a076-b39842f07f12\") " Oct 29 22:48:26.966010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.957658 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-certs" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:26.966010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.958279 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-socket" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:26.966010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.958794 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:26.966010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.959761 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-data" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:26.966010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.960274 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7635edde-2274-444f-a076-b39842f07f12-istiod-ca-cert" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:26.987436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.972497 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-credential-socket" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:26.993487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.992908 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" podUID="0b1ada16-8a79-4fd4-9320-e38c6f539e97" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": dial tcp 10.134.0.158:8000: connect: connection refused" Oct 29 22:48:26.993928 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:26.993880 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7635edde-2274-444f-a076-b39842f07f12-istio-podinfo" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.099521 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-istio-token" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.099596 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-kube-api-access-v6pj6" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "kube-api-access-v6pj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.101876 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7635edde-2274-444f-a076-b39842f07f12-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.101901 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.101924 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7635edde-2274-444f-a076-b39842f07f12-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.101938 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.101953 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6pj6\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-kube-api-access-v6pj6\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.101966 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.101980 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7635edde-2274-444f-a076-b39842f07f12-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.132759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.101994 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.137995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.137960 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-envoy" pod "7635edde-2274-444f-a076-b39842f07f12" (UID: "7635edde-2274-444f-a076-b39842f07f12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.165336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.165304 5610 scope.go:122] "RemoveContainer" containerID="8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b" Oct 29 22:48:27.167927 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:48:27.167897 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b\": container with ID starting with 8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b not found: ID does not exist" containerID="8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b" Oct 29 22:48:27.168141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.168116 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b"} err="failed to get container status \"8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b\": rpc error: code = NotFound desc = could not find container \"8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b\": container with ID starting with 8883da417f6e77ca326f979dfb2b82312943d2e5b40f2458ee9277eae5777f9b not found: ID does not exist" Oct 29 22:48:27.206443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.204033 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7635edde-2274-444f-a076-b39842f07f12-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.237195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.230097 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd"] Oct 29 22:48:27.237195 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.236952 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-gsqxd"] Oct 29 22:48:27.440882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.440853 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:48:27.477271 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.476915 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8"] Oct 29 22:48:27.613635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.613602 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1ada16-8a79-4fd4-9320-e38c6f539e97-kube-api-access-xnfp5\" (UniqueName: \"kubernetes.io/projected/0b1ada16-8a79-4fd4-9320-e38c6f539e97-kube-api-access-xnfp5\") pod \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " Oct 29 22:48:27.613800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.613782 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-dshm\") pod \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " Oct 29 22:48:27.613848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.613833 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tmp-dir\") pod \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " Oct 29 22:48:27.614047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.614027 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tls-certs\") pod \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " Oct 29 22:48:27.614292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.614105 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tmp-dir" pod "0b1ada16-8a79-4fd4-9320-e38c6f539e97" (UID: "0b1ada16-8a79-4fd4-9320-e38c6f539e97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.614581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.614557 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-home\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-home\") pod \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " Oct 29 22:48:27.614741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.614724 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-model-cache\") pod \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\" (UID: \"0b1ada16-8a79-4fd4-9320-e38c6f539e97\") " Oct 29 22:48:27.615023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.614971 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-model-cache" pod "0b1ada16-8a79-4fd4-9320-e38c6f539e97" (UID: "0b1ada16-8a79-4fd4-9320-e38c6f539e97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.615344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.615321 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-home" pod "0b1ada16-8a79-4fd4-9320-e38c6f539e97" (UID: "0b1ada16-8a79-4fd4-9320-e38c6f539e97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.615702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.615676 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.615791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.615707 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.615791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.615722 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.615878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.615863 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-dshm" pod "0b1ada16-8a79-4fd4-9320-e38c6f539e97" (UID: "0b1ada16-8a79-4fd4-9320-e38c6f539e97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:27.615970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.615949 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1ada16-8a79-4fd4-9320-e38c6f539e97-kube-api-access-xnfp5" pod "0b1ada16-8a79-4fd4-9320-e38c6f539e97" (UID: "0b1ada16-8a79-4fd4-9320-e38c6f539e97"). InnerVolumeSpecName "kube-api-access-xnfp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:27.616285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.616254 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tls-certs" pod "0b1ada16-8a79-4fd4-9320-e38c6f539e97" (UID: "0b1ada16-8a79-4fd4-9320-e38c6f539e97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:27.641188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.641150 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7635edde-2274-444f-a076-b39842f07f12" path="/var/lib/kubelet/pods/7635edde-2274-444f-a076-b39842f07f12/volumes" Oct 29 22:48:27.717186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.717148 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnfp5\" (UniqueName: \"kubernetes.io/projected/0b1ada16-8a79-4fd4-9320-e38c6f539e97-kube-api-access-xnfp5\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.717186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.717179 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1ada16-8a79-4fd4-9320-e38c6f539e97-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.717186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.717191 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1ada16-8a79-4fd4-9320-e38c6f539e97-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:27.876504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.876411 5610 generic.go:356] "Generic (PLEG): container finished" podID="0b1ada16-8a79-4fd4-9320-e38c6f539e97" containerID="4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b" exitCode=0 Oct 29 22:48:27.876504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.876450 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" Oct 29 22:48:27.876504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.876475 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" event={"ID":"0b1ada16-8a79-4fd4-9320-e38c6f539e97","Type":"ContainerDied","Data":"4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b"} Oct 29 22:48:27.876776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.876514 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp" event={"ID":"0b1ada16-8a79-4fd4-9320-e38c6f539e97","Type":"ContainerDied","Data":"55d44576d944e2ad43e3adf1fd3a58a3c9e93c9e1200c3897d4744f3c3ef226e"} Oct 29 22:48:27.876776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.876531 5610 scope.go:122] "RemoveContainer" containerID="4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b" Oct 29 22:48:27.878642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.878614 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" event={"ID":"bc290c4e-3d45-4547-bedc-ef18c50947b6","Type":"ContainerStarted","Data":"4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c"} Oct 29 22:48:27.878759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.878656 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" event={"ID":"bc290c4e-3d45-4547-bedc-ef18c50947b6","Type":"ContainerStarted","Data":"8b5e96a56634eef5420c630247733b2822455f695ccf85e87055f9ab15c57ccb"} Oct 29 22:48:27.887334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.887316 5610 scope.go:122] "RemoveContainer" containerID="4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b" Oct 29 22:48:27.887607 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:48:27.887588 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b\": container with ID starting with 4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b not found: ID does not exist" containerID="4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b" Oct 29 22:48:27.887686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.887614 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b"} err="failed to get container status \"4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b\": rpc error: code = NotFound desc = could not find container \"4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b\": container with ID starting with 4485c6256da4773e9180c4a7bcb43cb163820571f1b965bf8f0cdea0318f498b not found: ID does not exist" Oct 29 22:48:27.924887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.924863 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp"] Oct 29 22:48:27.932625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.932599 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a558b3/test-model-a558b3-kserve-757d5b65cf-v6zxp"] Oct 29 22:48:27.959714 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.959686 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:27.968712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:27.968664 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" podStartSLOduration=1.968649451 podStartE2EDuration="1.968649451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:27.962083619 +0000 UTC m=+1970.443401113" watchObservedRunningTime="2026-10-29 22:48:27.968649451 +0000 UTC m=+1970.449966988" Oct 29 22:48:28.745652 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:28.745591 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:29.638422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.638377 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1ada16-8a79-4fd4-9320-e38c6f539e97" path="/var/lib/kubelet/pods/0b1ada16-8a79-4fd4-9320-e38c6f539e97/volumes" Oct 29 22:48:29.973880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.973844 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj"] Oct 29 22:48:29.974815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.974796 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0b1ada16-8a79-4fd4-9320-e38c6f539e97" containerName="main" Oct 29 22:48:29.974895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.974818 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1ada16-8a79-4fd4-9320-e38c6f539e97" containerName="main" Oct 29 22:48:29.974895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.974839 5610 cpu_manager.go:414] "containerMap: removing container" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" Oct 29 22:48:29.974895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.974844 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" Oct 29 22:48:29.975053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.975011 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="7635edde-2274-444f-a076-b39842f07f12" containerName="istio-proxy" Oct 29 22:48:29.975053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.975034 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1ada16-8a79-4fd4-9320-e38c6f539e97" containerName="main" Oct 29 22:48:29.979388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:29.979363 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.006010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.003386 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj"] Oct 29 22:48:30.006010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.004858 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6bc7b2\"/\"test-model-6bc7b2-kserve-self-signed-certs\"" Oct 29 22:48:30.006010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.004970 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6bc7b2\"/\"default-dockercfg-gsrmv\"" Oct 29 22:48:30.006346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.006299 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6bc7b2\"/\"openshift-service-ca.crt\"" Oct 29 22:48:30.007080 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.006773 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6bc7b2\"/\"kube-root-ca.crt\"" Oct 29 22:48:30.011122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.011095 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:30.058730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.058699 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6wlp\" (UniqueName: \"kubernetes.io/projected/089f646b-d23c-481a-b1e3-19fce519cd91-kube-api-access-f6wlp\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.058924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.058765 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-dshm\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.058924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.058852 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089f646b-d23c-481a-b1e3-19fce519cd91-tls-certs\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.059078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.059058 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-home\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.059201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.059183 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-tmp-dir\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.059287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.059274 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-model-cache\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.152409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.152362 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:30.160349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.160322 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-tmp-dir\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.160523 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.160445 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-model-cache\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.160523 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.160494 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6wlp\" (UniqueName: \"kubernetes.io/projected/089f646b-d23c-481a-b1e3-19fce519cd91-kube-api-access-f6wlp\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.160653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.160552 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-dshm\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.160653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.160591 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089f646b-d23c-481a-b1e3-19fce519cd91-tls-certs\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.160753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.160739 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-home\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.160816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.160793 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-tmp-dir\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.160942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.160916 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-model-cache\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.161140 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.161121 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-home\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.163098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.163073 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-dshm\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.163491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.163472 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089f646b-d23c-481a-b1e3-19fce519cd91-tls-certs\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.177952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.177921 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6wlp\" (UniqueName: \"kubernetes.io/projected/089f646b-d23c-481a-b1e3-19fce519cd91-kube-api-access-f6wlp\") pod \"test-model-6bc7b2-kserve-769bd6fd79-r5kbj\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.300475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.298515 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:30.303959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.300873 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:30.732224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.732198 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj"] Oct 29 22:48:30.732954 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:48:30.732930 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod089f646b_d23c_481a_b1e3_19fce519cd91.slice/crio-5877e29c5c6f4b3b67838334afe2f86df6a271971a043da106d4b4dd2d19764d WatchSource:0}: Error finding container 5877e29c5c6f4b3b67838334afe2f86df6a271971a043da106d4b4dd2d19764d: Status 404 returned error can't find the container with id 5877e29c5c6f4b3b67838334afe2f86df6a271971a043da106d4b4dd2d19764d Oct 29 22:48:30.909873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.909840 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" event={"ID":"089f646b-d23c-481a-b1e3-19fce519cd91","Type":"ContainerStarted","Data":"c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727"} Oct 29 22:48:30.910052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.909884 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" event={"ID":"089f646b-d23c-481a-b1e3-19fce519cd91","Type":"ContainerStarted","Data":"5877e29c5c6f4b3b67838334afe2f86df6a271971a043da106d4b4dd2d19764d"} Oct 29 22:48:30.946640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:30.946583 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" podStartSLOduration=1.946567853 podStartE2EDuration="1.946567853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:30.944048886 +0000 UTC m=+1973.425366378" watchObservedRunningTime="2026-10-29 22:48:30.946567853 +0000 UTC m=+1973.427885346" Oct 29 22:48:31.320844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:31.320806 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:31.951055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:31.936423 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:32.777203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.777169 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2"] Oct 29 22:48:32.782302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.782277 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.797750 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.797617 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8dd4c8\"/\"kube-root-ca.crt\"" Oct 29 22:48:32.807726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.807698 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8dd4c8\"/\"openshift-service-ca.crt\"" Oct 29 22:48:32.808680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.808648 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8dd4c8\"/\"test-model-8dd4c8-kserve-self-signed-certs\"" Oct 29 22:48:32.808815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.808725 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8dd4c8\"/\"default-dockercfg-gjjwg\"" Oct 29 22:48:32.810035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.810002 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2"] Oct 29 22:48:32.824504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.824477 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tls-certs\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.824695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.824675 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-home\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.824772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.824733 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-model-cache\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.824843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.824827 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvms\" (UniqueName: \"kubernetes.io/projected/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-kube-api-access-tfvms\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.824930 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.824916 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-dshm\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.825019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.824994 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tmp-dir\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.873809 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.873772 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:32.926034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.925998 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-home\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.926240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.926054 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-model-cache\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.926240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.926075 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvms\" (UniqueName: \"kubernetes.io/projected/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-kube-api-access-tfvms\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.926240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.926128 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-dshm\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.926240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.926147 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tmp-dir\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.926540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.926292 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tls-certs\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.926540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.926454 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-home\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.926540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.926528 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-model-cache\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.926663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.926630 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tmp-dir\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.928854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.928823 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-dshm\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.929019 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.929000 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tls-certs\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:32.943988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:32.943966 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvms\" (UniqueName: \"kubernetes.io/projected/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-kube-api-access-tfvms\") pod \"test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:33.104688 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:33.104598 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:33.530071 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:48:33.530042 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad9976c_7a0c_468b_b3f3_16fc3a567eda.slice/crio-38d9bddbe1f81e28d19d2442b72f40567776bc56e0d1570f879bf892f8d95109 WatchSource:0}: Error finding container 38d9bddbe1f81e28d19d2442b72f40567776bc56e0d1570f879bf892f8d95109: Status 404 returned error can't find the container with id 38d9bddbe1f81e28d19d2442b72f40567776bc56e0d1570f879bf892f8d95109 Oct 29 22:48:33.530378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:33.530351 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2"] Oct 29 22:48:33.612569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:33.604995 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:33.990571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:33.990524 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" event={"ID":"7ad9976c-7a0c-468b-b3f3-16fc3a567eda","Type":"ContainerStarted","Data":"0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1"} Oct 29 22:48:33.990571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:33.990585 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" event={"ID":"7ad9976c-7a0c-468b-b3f3-16fc3a567eda","Type":"ContainerStarted","Data":"38d9bddbe1f81e28d19d2442b72f40567776bc56e0d1570f879bf892f8d95109"} Oct 29 22:48:34.029026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:34.028971 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" podStartSLOduration=2.028955873 podStartE2EDuration="2.028955873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:34.025316416 +0000 UTC m=+1976.506633910" watchObservedRunningTime="2026-10-29 22:48:34.028955873 +0000 UTC m=+1976.510273403" Oct 29 22:48:34.105725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:34.105689 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:35.066243 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:35.066206 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:36.684090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:36.684053 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:36.959838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:36.959803 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:36.973006 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:36.972975 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:37.000153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:37.000119 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf"] Oct 29 22:48:37.024546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:37.024519 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk"] Oct 29 22:48:37.032471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:37.032448 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:48:37.182870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:37.182832 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:37.330373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:37.330275 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:38.311334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:38.311290 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:38.965455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:38.965388 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:39.910407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:39.908060 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:40.320911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:40.320882 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:40.333380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:40.333353 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:41.062949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:41.062919 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:48:41.547208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:41.547172 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:42.001284 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.001237 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" podUID="1336dcd8-9214-4694-b955-c3fd3cc865ec" containerName="payload-processing" containerID="cri-o://08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547" gracePeriod=25 Oct 29 22:48:42.025253 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.025195 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" podUID="73b700f4-19cf-4aaa-885a-f328c224d5c4" containerName="payload-pre-processing" containerID="cri-o://20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2" gracePeriod=25 Oct 29 22:48:42.338059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.338030 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:48:42.369989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.369955 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1336dcd8-9214-4694-b955-c3fd3cc865ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1336dcd8-9214-4694-b955-c3fd3cc865ec-plugins-config-volume\") pod \"1336dcd8-9214-4694-b955-c3fd3cc865ec\" (UID: \"1336dcd8-9214-4694-b955-c3fd3cc865ec\") " Oct 29 22:48:42.370222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.370202 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1336dcd8-9214-4694-b955-c3fd3cc865ec-kube-api-access-v62lc\" (UniqueName: \"kubernetes.io/projected/1336dcd8-9214-4694-b955-c3fd3cc865ec-kube-api-access-v62lc\") pod \"1336dcd8-9214-4694-b955-c3fd3cc865ec\" (UID: \"1336dcd8-9214-4694-b955-c3fd3cc865ec\") " Oct 29 22:48:42.370347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.370321 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1336dcd8-9214-4694-b955-c3fd3cc865ec-plugins-config-volume" pod "1336dcd8-9214-4694-b955-c3fd3cc865ec" (UID: "1336dcd8-9214-4694-b955-c3fd3cc865ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:42.370994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.370962 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1336dcd8-9214-4694-b955-c3fd3cc865ec-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:42.372471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.372442 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1336dcd8-9214-4694-b955-c3fd3cc865ec-kube-api-access-v62lc" pod "1336dcd8-9214-4694-b955-c3fd3cc865ec" (UID: "1336dcd8-9214-4694-b955-c3fd3cc865ec"). InnerVolumeSpecName "kube-api-access-v62lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:42.473145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.473112 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v62lc\" (UniqueName: \"kubernetes.io/projected/1336dcd8-9214-4694-b955-c3fd3cc865ec-kube-api-access-v62lc\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:42.729194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.729154 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm"] Oct 29 22:48:42.729475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.729445 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" podUID="f52d3f42-c8b7-4377-a22c-921b0b66e237" containerName="maas-api" containerID="cri-o://e6b1cc0f13dec373fd388a6bfee48620df7ab14e542d0a8b69d0cacf29120d1f" gracePeriod=30 Oct 29 22:48:42.851183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:42.851150 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:43.147854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.145507 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:43.210791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.210736 5610 generic.go:356] "Generic (PLEG): container finished" podID="1336dcd8-9214-4694-b955-c3fd3cc865ec" containerID="08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547" exitCode=0 Oct 29 22:48:43.211252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.211205 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" event={"ID":"1336dcd8-9214-4694-b955-c3fd3cc865ec","Type":"ContainerDied","Data":"08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547"} Oct 29 22:48:43.211503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.211465 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" event={"ID":"1336dcd8-9214-4694-b955-c3fd3cc865ec","Type":"ContainerDied","Data":"8ffa33cbadbc8cbc0e2f61f4c293d678908f4fd53c56f87d6c37dc92a0068b79"} Oct 29 22:48:43.211674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.211659 5610 scope.go:122] "RemoveContainer" containerID="08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547" Oct 29 22:48:43.212085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.212061 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf" Oct 29 22:48:43.493472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.438077 5610 generic.go:356] "Generic (PLEG): container finished" podID="f52d3f42-c8b7-4377-a22c-921b0b66e237" containerID="e6b1cc0f13dec373fd388a6bfee48620df7ab14e542d0a8b69d0cacf29120d1f" exitCode=0 Oct 29 22:48:43.493472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.438303 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" event={"ID":"f52d3f42-c8b7-4377-a22c-921b0b66e237","Type":"ContainerDied","Data":"e6b1cc0f13dec373fd388a6bfee48620df7ab14e542d0a8b69d0cacf29120d1f"} Oct 29 22:48:43.493472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.465145 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:43.493472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.486915 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:48:43.535424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.523861 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:43.552586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.552556 5610 scope.go:122] "RemoveContainer" containerID="08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547" Oct 29 22:48:43.553323 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:48:43.553296 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547\": container with ID starting with 08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547 not found: ID does not exist" containerID="08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547" Oct 29 22:48:43.553465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.553334 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547"} err="failed to get container status \"08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547\": rpc error: code = NotFound desc = could not find container \"08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547\": container with ID starting with 08ee15362ddd212e86875fca19d69036a4837403907d29f5d24636f5ae8f4547 not found: ID does not exist" Oct 29 22:48:43.587607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.587573 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-metrics-certs\") pod \"f52d3f42-c8b7-4377-a22c-921b0b66e237\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " Oct 29 22:48:43.587821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.587686 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52d3f42-c8b7-4377-a22c-921b0b66e237-kube-api-access-pw62c\" (UniqueName: \"kubernetes.io/projected/f52d3f42-c8b7-4377-a22c-921b0b66e237-kube-api-access-pw62c\") pod \"f52d3f42-c8b7-4377-a22c-921b0b66e237\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " Oct 29 22:48:43.587821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.587710 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-maas-api-tls\") pod \"f52d3f42-c8b7-4377-a22c-921b0b66e237\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " Oct 29 22:48:43.587821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.587766 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52d3f42-c8b7-4377-a22c-921b0b66e237-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52d3f42-c8b7-4377-a22c-921b0b66e237-openshift-service-ca\") pod \"f52d3f42-c8b7-4377-a22c-921b0b66e237\" (UID: \"f52d3f42-c8b7-4377-a22c-921b0b66e237\") " Oct 29 22:48:43.588912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.588883 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52d3f42-c8b7-4377-a22c-921b0b66e237-openshift-service-ca" pod "f52d3f42-c8b7-4377-a22c-921b0b66e237" (UID: "f52d3f42-c8b7-4377-a22c-921b0b66e237"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:43.591324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.591233 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-maas-api-tls" pod "f52d3f42-c8b7-4377-a22c-921b0b66e237" (UID: "f52d3f42-c8b7-4377-a22c-921b0b66e237"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:43.591324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.591285 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-metrics-certs" pod "f52d3f42-c8b7-4377-a22c-921b0b66e237" (UID: "f52d3f42-c8b7-4377-a22c-921b0b66e237"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:48:43.591586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.591565 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52d3f42-c8b7-4377-a22c-921b0b66e237-kube-api-access-pw62c" pod "f52d3f42-c8b7-4377-a22c-921b0b66e237" (UID: "f52d3f42-c8b7-4377-a22c-921b0b66e237"). InnerVolumeSpecName "kube-api-access-pw62c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:43.618576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.609733 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf"] Oct 29 22:48:43.618576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.615026 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a558b3-688cbd55dd-95wlf"] Oct 29 22:48:43.638594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.638563 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1336dcd8-9214-4694-b955-c3fd3cc865ec" path="/var/lib/kubelet/pods/1336dcd8-9214-4694-b955-c3fd3cc865ec/volumes" Oct 29 22:48:43.664871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.664642 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:43.703105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.693940 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.703105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.693978 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw62c\" (UniqueName: \"kubernetes.io/projected/f52d3f42-c8b7-4377-a22c-921b0b66e237-kube-api-access-pw62c\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.703105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.693993 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f52d3f42-c8b7-4377-a22c-921b0b66e237-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.703105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.694009 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52d3f42-c8b7-4377-a22c-921b0b66e237-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:43.797494 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:48:43.796610 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:48:43.823335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:43.823287 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:44.117629 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:44.117545 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:48:44.210831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:44.210791 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:44.214875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:44.214838 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" podUID="73b700f4-19cf-4aaa-885a-f328c224d5c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:44.451222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:44.451197 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" Oct 29 22:48:44.451440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:44.451243 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm" event={"ID":"f52d3f42-c8b7-4377-a22c-921b0b66e237","Type":"ContainerDied","Data":"5b5eb0a6737fa6c88eb4d99d6ca064c7731d68df69973ab90c249a76459c6d98"} Oct 29 22:48:44.451440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:44.451321 5610 scope.go:122] "RemoveContainer" containerID="e6b1cc0f13dec373fd388a6bfee48620df7ab14e542d0a8b69d0cacf29120d1f" Oct 29 22:48:44.500535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:44.500502 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm"] Oct 29 22:48:44.511461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:44.511433 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a558b3-68d655d769-dmtpm"] Oct 29 22:48:45.585424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:45.579075 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:45.638564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:45.638533 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52d3f42-c8b7-4377-a22c-921b0b66e237" path="/var/lib/kubelet/pods/f52d3f42-c8b7-4377-a22c-921b0b66e237/volumes" Oct 29 22:48:46.951789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:46.951756 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:48.910382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:48.910334 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:50.674469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:50.673828 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:51.296320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:51.296277 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:51.483222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:51.483170 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:51.661223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:51.660482 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg"] Oct 29 22:48:51.661223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:51.660740 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" containerID="cri-o://a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb" gracePeriod=30 Oct 29 22:48:51.686188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:51.686155 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:52.935420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.935375 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr"] Oct 29 22:48:52.936302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.936283 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f52d3f42-c8b7-4377-a22c-921b0b66e237" containerName="maas-api" Oct 29 22:48:52.936302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.936302 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52d3f42-c8b7-4377-a22c-921b0b66e237" containerName="maas-api" Oct 29 22:48:52.936492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.936332 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1336dcd8-9214-4694-b955-c3fd3cc865ec" containerName="payload-processing" Oct 29 22:48:52.936492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.936337 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1336dcd8-9214-4694-b955-c3fd3cc865ec" containerName="payload-processing" Oct 29 22:48:52.936492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.936470 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52d3f42-c8b7-4377-a22c-921b0b66e237" containerName="maas-api" Oct 29 22:48:52.936492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.936482 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1336dcd8-9214-4694-b955-c3fd3cc865ec" containerName="payload-processing" Oct 29 22:48:52.941846 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.941829 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:52.954382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.954359 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-57d336-serving-cert\"" Oct 29 22:48:52.970879 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:52.970842 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr"] Oct 29 22:48:53.046954 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.046919 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-metrics-certs\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.047224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.047203 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a7170c-6579-469d-a4ed-deb08c89a86f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.047278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.047247 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4tp9\" (UniqueName: \"kubernetes.io/projected/84a7170c-6579-469d-a4ed-deb08c89a86f-kube-api-access-c4tp9\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.047362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.047336 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-maas-api-tls\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.071598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.071564 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:53.149386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.149352 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a7170c-6579-469d-a4ed-deb08c89a86f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.149566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.149416 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4tp9\" (UniqueName: \"kubernetes.io/projected/84a7170c-6579-469d-a4ed-deb08c89a86f-kube-api-access-c4tp9\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.149566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.149449 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-maas-api-tls\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.149566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.149554 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-metrics-certs\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.150162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.150132 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a7170c-6579-469d-a4ed-deb08c89a86f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.152125 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.152099 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-metrics-certs\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.152299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.152281 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-maas-api-tls\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.167089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.167067 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4tp9\" (UniqueName: \"kubernetes.io/projected/84a7170c-6579-469d-a4ed-deb08c89a86f-kube-api-access-c4tp9\") pod \"maas-api-e2e-inf-b-57d336-664c59b858-tfgsr\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.220053 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.218751 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:53.293831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.292348 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:53.613941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:53.613910 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr"] Oct 29 22:48:53.614257 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:48:53.614229 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a7170c_6579_469d_a4ed_deb08c89a86f.slice/crio-4d5fea59eaf79d19694e21ed33c5511203e9446aea18794e3c59d1e9a96b87e4 WatchSource:0}: Error finding container 4d5fea59eaf79d19694e21ed33c5511203e9446aea18794e3c59d1e9a96b87e4: Status 404 returned error can't find the container with id 4d5fea59eaf79d19694e21ed33c5511203e9446aea18794e3c59d1e9a96b87e4 Oct 29 22:48:54.278475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:54.273381 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" podUID="73b700f4-19cf-4aaa-885a-f328c224d5c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:48:54.608673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:54.608573 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" event={"ID":"84a7170c-6579-469d-a4ed-deb08c89a86f","Type":"ContainerStarted","Data":"498dd05a0729b38bdcc29a0fde9edc6a4284b00f1263dbb574882ed565511d82"} Oct 29 22:48:54.608856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:54.608690 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" event={"ID":"84a7170c-6579-469d-a4ed-deb08c89a86f","Type":"ContainerStarted","Data":"4d5fea59eaf79d19694e21ed33c5511203e9446aea18794e3c59d1e9a96b87e4"} Oct 29 22:48:54.610259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:54.610229 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:48:54.669248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:54.669183 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" podStartSLOduration=2.292633808 podStartE2EDuration="2.669164483s" totalImagesPullingTime="376.530675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:48:54.66791103 +0000 UTC m=+1997.149228523" watchObservedRunningTime="2026-10-29 22:48:54.669164483 +0000 UTC m=+1997.150481977" Oct 29 22:48:55.133674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:55.133642 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:56.730485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:56.729907 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:57.017974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.017948 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:48:57.181564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.181528 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-podinfo\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.181771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.181639 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-certs\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.181771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.181714 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-kube-api-access-nchj9\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-kube-api-access-nchj9\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.181910 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.181860 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-credential-socket\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.181968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.181952 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-certs" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:57.182023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.181979 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-envoy\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.182023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.182007 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-socket\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.182122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.182105 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-token\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-token\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.182182 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.182165 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-data\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.182272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.182237 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974c4bbd-66b9-46ff-9533-ee5c20c71313-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974c4bbd-66b9-46ff-9533-ee5c20c71313-istiod-ca-cert\") pod \"974c4bbd-66b9-46ff-9533-ee5c20c71313\" (UID: \"974c4bbd-66b9-46ff-9533-ee5c20c71313\") " Oct 29 22:48:57.182340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.182100 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-credential-socket" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:57.182340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.182303 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-socket" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:57.182560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.182537 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-data" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:57.182837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.182814 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974c4bbd-66b9-46ff-9533-ee5c20c71313-istiod-ca-cert" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:48:57.183183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.183166 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.183296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.183187 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.183296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.183202 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.183296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.183216 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.183296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.183229 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/974c4bbd-66b9-46ff-9533-ee5c20c71313-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.184499 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.184459 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-kube-api-access-nchj9" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "kube-api-access-nchj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:57.184600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.184514 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-podinfo" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:48:57.184711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.184688 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-envoy" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:48:57.184900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.184872 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-token" pod "974c4bbd-66b9-46ff-9533-ee5c20c71313" (UID: "974c4bbd-66b9-46ff-9533-ee5c20c71313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:48:57.284814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.284723 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.284814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.284752 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.284814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.284764 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nchj9\" (UniqueName: \"kubernetes.io/projected/974c4bbd-66b9-46ff-9533-ee5c20c71313-kube-api-access-nchj9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.284814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.284774 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/974c4bbd-66b9-46ff-9533-ee5c20c71313-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:48:57.637147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.637070 5610 generic.go:356] "Generic (PLEG): container finished" podID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerID="a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb" exitCode=0 Oct 29 22:48:57.637297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.637174 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" Oct 29 22:48:57.638816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.638794 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" event={"ID":"974c4bbd-66b9-46ff-9533-ee5c20c71313","Type":"ContainerDied","Data":"a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb"} Oct 29 22:48:57.638903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.638835 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg" event={"ID":"974c4bbd-66b9-46ff-9533-ee5c20c71313","Type":"ContainerDied","Data":"3e28e5b058334156f72dc93edd6cac8b64a20a3adac98cee4cbf4e15453a51de"} Oct 29 22:48:57.638903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.638858 5610 scope.go:122] "RemoveContainer" containerID="a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb" Oct 29 22:48:57.651541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.651519 5610 scope.go:122] "RemoveContainer" containerID="d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2" Oct 29 22:48:57.662458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.662440 5610 scope.go:122] "RemoveContainer" containerID="a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb" Oct 29 22:48:57.662787 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:48:57.662765 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb\": container with ID starting with a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb not found: ID does not exist" containerID="a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb" Oct 29 22:48:57.662876 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.662796 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb"} err="failed to get container status \"a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb\": rpc error: code = NotFound desc = could not find container \"a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb\": container with ID starting with a287ca3aad63cc12c721268106c024dbd813899e74fe3eb2b926b6d9732eeebb not found: ID does not exist" Oct 29 22:48:57.662876 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.662822 5610 scope.go:122] "RemoveContainer" containerID="d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2" Oct 29 22:48:57.663098 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:48:57.663077 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2\": container with ID starting with d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2 not found: ID does not exist" containerID="d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2" Oct 29 22:48:57.663147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.663107 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2"} err="failed to get container status \"d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2\": rpc error: code = NotFound desc = could not find container \"d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2\": container with ID starting with d0ebda7c051c089637ee6db49f8f0e5b53021c63c06fb04322073371dfe83bc2 not found: ID does not exist" Oct 29 22:48:57.683942 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.683916 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg"] Oct 29 22:48:57.694202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:57.694173 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a558b3-openshift-default-8476c499c5-jjjvg"] Oct 29 22:48:58.169170 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:58.169137 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:48:59.638807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:59.638772 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" path="/var/lib/kubelet/pods/974c4bbd-66b9-46ff-9533-ee5c20c71313/volumes" Oct 29 22:48:59.867413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:48:59.863637 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:00.519689 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:00.519655 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:00.664498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:00.664463 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:01.449927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:01.449891 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:01.637964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:01.637934 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:49:01.772926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:01.772844 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:02.349854 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.349827 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:49:02.375267 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.375243 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b700f4-19cf-4aaa-885a-f328c224d5c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b700f4-19cf-4aaa-885a-f328c224d5c4-plugins-config-volume\") pod \"73b700f4-19cf-4aaa-885a-f328c224d5c4\" (UID: \"73b700f4-19cf-4aaa-885a-f328c224d5c4\") " Oct 29 22:49:02.375429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.375281 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b700f4-19cf-4aaa-885a-f328c224d5c4-kube-api-access-qwdxx\" (UniqueName: \"kubernetes.io/projected/73b700f4-19cf-4aaa-885a-f328c224d5c4-kube-api-access-qwdxx\") pod \"73b700f4-19cf-4aaa-885a-f328c224d5c4\" (UID: \"73b700f4-19cf-4aaa-885a-f328c224d5c4\") " Oct 29 22:49:02.375622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.375599 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b700f4-19cf-4aaa-885a-f328c224d5c4-plugins-config-volume" pod "73b700f4-19cf-4aaa-885a-f328c224d5c4" (UID: "73b700f4-19cf-4aaa-885a-f328c224d5c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:49:02.375973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.375953 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b700f4-19cf-4aaa-885a-f328c224d5c4-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:02.377540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.377514 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b700f4-19cf-4aaa-885a-f328c224d5c4-kube-api-access-qwdxx" pod "73b700f4-19cf-4aaa-885a-f328c224d5c4" (UID: "73b700f4-19cf-4aaa-885a-f328c224d5c4"). InnerVolumeSpecName "kube-api-access-qwdxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:02.477067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.477036 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdxx\" (UniqueName: \"kubernetes.io/projected/73b700f4-19cf-4aaa-885a-f328c224d5c4-kube-api-access-qwdxx\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:02.586265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.586168 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh"] Oct 29 22:49:02.635508 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.635461 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg"] Oct 29 22:49:02.698424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.697893 5610 generic.go:356] "Generic (PLEG): container finished" podID="73b700f4-19cf-4aaa-885a-f328c224d5c4" containerID="20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2" exitCode=0 Oct 29 22:49:02.698424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.698149 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" event={"ID":"73b700f4-19cf-4aaa-885a-f328c224d5c4","Type":"ContainerDied","Data":"20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2"} Oct 29 22:49:02.698424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.698196 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" event={"ID":"73b700f4-19cf-4aaa-885a-f328c224d5c4","Type":"ContainerDied","Data":"60a6a5e8a06223b89864096715515663b85ae575c8aba31c8abaf426e26a11f1"} Oct 29 22:49:02.698424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.698225 5610 scope.go:122] "RemoveContainer" containerID="20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2" Oct 29 22:49:02.698424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.698363 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk" Oct 29 22:49:02.728680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.728651 5610 scope.go:122] "RemoveContainer" containerID="20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2" Oct 29 22:49:02.729124 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:49:02.729102 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2\": container with ID starting with 20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2 not found: ID does not exist" containerID="20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2" Oct 29 22:49:02.729215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.729136 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2"} err="failed to get container status \"20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2\": rpc error: code = NotFound desc = could not find container \"20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2\": container with ID starting with 20e116100a33ae1713c82cb37fbd86958b54254b961b2070fd8bb8f8ba9daad2 not found: ID does not exist" Oct 29 22:49:02.779522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.779482 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk"] Oct 29 22:49:02.785199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.785166 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:02.788080 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.788059 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a558b3-d6dc777b7-xxfvk"] Oct 29 22:49:02.931473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:02.931318 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:03.067883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:03.067840 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:03.638694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:03.638663 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b700f4-19cf-4aaa-885a-f328c224d5c4" path="/var/lib/kubelet/pods/73b700f4-19cf-4aaa-885a-f328c224d5c4/volumes" Oct 29 22:49:04.889418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:04.889228 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:06.769169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:06.769123 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:07.587418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:07.587102 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" podUID="e92999ac-9100-4f7e-9010-f9f720d5e4a7" containerName="payload-processing" containerID="cri-o://e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f" gracePeriod=25 Oct 29 22:49:07.636969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:07.636907 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" podUID="1530f5dc-2049-48f7-aef2-55d5e5956f5f" containerName="payload-pre-processing" containerID="cri-o://fd75b80eb24ff752a93de1a80bb8542190c904c2c3bb45828c34490d66995cfa" gracePeriod=25 Oct 29 22:49:08.536799 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:08.536753 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:08.682413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:08.682067 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:10.158416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:10.155483 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:11.614061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:11.614022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:12.929633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.929595 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv"] Oct 29 22:49:12.930520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.930499 5610 cpu_manager.go:414] "containerMap: removing container" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" Oct 29 22:49:12.930520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.930518 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" Oct 29 22:49:12.930718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.930548 5610 cpu_manager.go:414] "containerMap: removing container" podUID="73b700f4-19cf-4aaa-885a-f328c224d5c4" containerName="payload-pre-processing" Oct 29 22:49:12.930718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.930553 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b700f4-19cf-4aaa-885a-f328c224d5c4" containerName="payload-pre-processing" Oct 29 22:49:12.930718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.930718 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b700f4-19cf-4aaa-885a-f328c224d5c4" containerName="payload-pre-processing" Oct 29 22:49:12.930901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.930733 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" Oct 29 22:49:12.930901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.930743 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" Oct 29 22:49:12.957633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.957602 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv"] Oct 29 22:49:12.957776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.957720 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:12.981119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.981089 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-437981\"/\"openshift-service-ca.crt\"" Oct 29 22:49:12.981305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.981089 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-437981\"/\"auth-test-model-437981-kserve-self-signed-certs\"" Oct 29 22:49:12.981383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.981224 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-437981\"/\"default-dockercfg-knfgv\"" Oct 29 22:49:12.981383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:12.981228 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-437981\"/\"kube-root-ca.crt\"" Oct 29 22:49:13.032093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.032056 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:13.138249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.138211 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-model-cache\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.138467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.138388 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-home\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.138557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.138464 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m54gp\" (UniqueName: \"kubernetes.io/projected/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-kube-api-access-m54gp\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.138557 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.138530 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tmp-dir\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.138682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.138605 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tls-certs\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.138682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.138640 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-dshm\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.241637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.240150 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-model-cache\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.241637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.240262 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-home\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.241637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.240315 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m54gp\" (UniqueName: \"kubernetes.io/projected/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-kube-api-access-m54gp\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.241637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.240381 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tmp-dir\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.241637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.240465 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tls-certs\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.241637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.240494 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-dshm\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.242927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.242897 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-model-cache\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.254440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.254009 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tmp-dir\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.274420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.254892 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-home\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.275155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.259795 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-dshm\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.291481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.284043 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m54gp\" (UniqueName: \"kubernetes.io/projected/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-kube-api-access-m54gp\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.301295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.301263 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tls-certs\") pod \"auth-test-model-437981-kserve-7d6cc8f64f-gbkfv\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.354005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.353882 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:13.581660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.581568 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:13.724880 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:49:13.724798 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:49:13.889231 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:49:13.889201 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e99b243_8803_4980_ab78_cc9c9a1ff7d0.slice/crio-abe0ad381660a5b4e02bce25a6ce62cee0b1f14a912f319b06640af8facb4c95 WatchSource:0}: Error finding container abe0ad381660a5b4e02bce25a6ce62cee0b1f14a912f319b06640af8facb4c95: Status 404 returned error can't find the container with id abe0ad381660a5b4e02bce25a6ce62cee0b1f14a912f319b06640af8facb4c95 Oct 29 22:49:13.891034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:13.891002 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv"] Oct 29 22:49:14.808608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.808496 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:14.869076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.869037 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" event={"ID":"0e99b243-8803-4980-ab78-cc9c9a1ff7d0","Type":"ContainerStarted","Data":"e7452dc946f63b006ce9d127af430e86a71824e49808cf9311508c404bf442be"} Oct 29 22:49:14.869264 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.869085 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" event={"ID":"0e99b243-8803-4980-ab78-cc9c9a1ff7d0","Type":"ContainerStarted","Data":"abe0ad381660a5b4e02bce25a6ce62cee0b1f14a912f319b06640af8facb4c95"} Oct 29 22:49:14.870873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.870847 5610 generic.go:356] "Generic (PLEG): container finished" podID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerID="054f76c1d32023c5b85579c7f24d44490023beec38f08a788c78e17a0bca6279" exitCode=0 Oct 29 22:49:14.871002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.870918 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" event={"ID":"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685","Type":"ContainerDied","Data":"054f76c1d32023c5b85579c7f24d44490023beec38f08a788c78e17a0bca6279"} Oct 29 22:49:14.871308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.871293 5610 scope.go:122] "RemoveContainer" containerID="054f76c1d32023c5b85579c7f24d44490023beec38f08a788c78e17a0bca6279" Oct 29 22:49:14.873196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.873165 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:49:14.873313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.873253 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:49:14.873313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.873301 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:49:14.907128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:14.907089 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" podStartSLOduration=2.907077114 podStartE2EDuration="2.907077114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:14.903031367 +0000 UTC m=+2017.384348862" watchObservedRunningTime="2026-10-29 22:49:14.907077114 +0000 UTC m=+2017.388394604" Oct 29 22:49:15.444420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:15.443484 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:15.582276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:15.582244 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:15.608827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:15.608785 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:15.849468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:15.846592 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:15.890322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:15.890283 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" event={"ID":"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685","Type":"ContainerStarted","Data":"ddfee9d5824929e70f17998ede3702e0e1bb213864b863c143b2e72a934de0cf"} Oct 29 22:49:16.045631 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:16.045590 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" podUID="1530f5dc-2049-48f7-aef2-55d5e5956f5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:49:16.045876 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:16.045734 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" podUID="e92999ac-9100-4f7e-9010-f9f720d5e4a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:49:16.109506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:16.109379 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:16.153205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:16.153176 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:49:16.153479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:16.153449 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:49:16.153540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:16.153514 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:49:17.153138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:17.153107 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:49:17.153549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:17.153174 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:49:17.675418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:17.673840 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:18.154074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:18.154036 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:49:18.154453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:18.154090 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:49:19.153509 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:19.153476 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:49:19.153718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:19.153532 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:49:19.182418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:19.179205 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:20.154108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:20.154067 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:49:20.154317 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:20.154122 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:49:21.161435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:21.158973 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:21.166162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:21.166131 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 29 22:49:21.166532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:21.166483 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 29 22:49:22.159255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:22.159224 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:49:22.159789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:22.159762 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:49:22.160599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:22.160581 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 22:49:23.067136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:23.067053 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:23.582409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:23.582360 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:23.597995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:23.597953 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:23.984061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:23.984030 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:49:24.068051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.068006 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:24.236363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.236275 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:24.320372 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.320341 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd"] Oct 29 22:49:24.320702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.320670 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" containerID="cri-o://e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054" gracePeriod=30 Oct 29 22:49:24.370874 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.370836 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r"] Oct 29 22:49:24.372175 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.372149 5610 cpu_manager.go:414] "containerMap: removing container" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" Oct 29 22:49:24.372175 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.372173 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974c4bbd-66b9-46ff-9533-ee5c20c71313" containerName="istio-proxy" Oct 29 22:49:24.376504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.376481 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.416029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.395253 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-84304f-openshift-default-dockercfg-p57fc\"" Oct 29 22:49:24.467414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.465654 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:24.534422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.531488 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r"] Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593321 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593411 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593509 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593545 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wv5\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-kube-api-access-n9wv5\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593601 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593688 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593735 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593799 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.594423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.593943 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13e838a-a7be-4b74-8918-5549a1681ba9-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.700919 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.700980 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.701056 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.701204 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13e838a-a7be-4b74-8918-5549a1681ba9-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.701296 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.701356 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.701471 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.701506 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wv5\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-kube-api-access-n9wv5\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.701576 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.704237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.702164 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.720414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.719489 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-credential-socket\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.720414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.719757 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-data\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.720414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.719914 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-certs\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.726586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.726554 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13e838a-a7be-4b74-8918-5549a1681ba9-istiod-ca-cert\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.732430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.732096 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-envoy\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.732869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.732847 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-podinfo\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.743173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.743143 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-token\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.743637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.743615 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wv5\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-kube-api-access-n9wv5\") pod \"e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:24.978420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:24.977857 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:25.014467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:25.013741 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:25.367284 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:25.367250 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r"] Oct 29 22:49:25.368665 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:49:25.368635 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13e838a_a7be_4b74_8918_5549a1681ba9.slice/crio-519fe5e712846e01f06b53364549810d1323a446dadf974ead9cab028335c1bf WatchSource:0}: Error finding container 519fe5e712846e01f06b53364549810d1323a446dadf974ead9cab028335c1bf: Status 404 returned error can't find the container with id 519fe5e712846e01f06b53364549810d1323a446dadf974ead9cab028335c1bf Oct 29 22:49:25.373743 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:25.373700 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:49:25.373868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:25.373851 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:49:25.373940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:25.373897 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:49:26.039970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:26.039935 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" event={"ID":"c13e838a-a7be-4b74-8918-5549a1681ba9","Type":"ContainerStarted","Data":"b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65"} Oct 29 22:49:26.040157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:26.039990 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" event={"ID":"c13e838a-a7be-4b74-8918-5549a1681ba9","Type":"ContainerStarted","Data":"519fe5e712846e01f06b53364549810d1323a446dadf974ead9cab028335c1bf"} Oct 29 22:49:26.045346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:26.045309 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" podUID="e92999ac-9100-4f7e-9010-f9f720d5e4a7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:49:26.045346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:26.045324 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" podUID="1530f5dc-2049-48f7-aef2-55d5e5956f5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:49:26.095442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:26.095367 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podStartSLOduration=2.095349025 podStartE2EDuration="2.095349025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:49:26.087559391 +0000 UTC m=+2028.568876883" watchObservedRunningTime="2026-10-29 22:49:26.095349025 +0000 UTC m=+2028.576666511" Oct 29 22:49:26.637777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:26.637744 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:27.017967 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:27.017934 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:27.019798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:27.019769 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:49:27.019956 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:27.019819 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:49:28.022438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.018426 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:49:28.022438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.018472 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:49:28.030358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.030334 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:49:28.101417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.095299 5610 generic.go:356] "Generic (PLEG): container finished" podID="1530f5dc-2049-48f7-aef2-55d5e5956f5f" containerID="fd75b80eb24ff752a93de1a80bb8542190c904c2c3bb45828c34490d66995cfa" exitCode=0 Oct 29 22:49:28.101417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.095518 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" event={"ID":"1530f5dc-2049-48f7-aef2-55d5e5956f5f","Type":"ContainerDied","Data":"fd75b80eb24ff752a93de1a80bb8542190c904c2c3bb45828c34490d66995cfa"} Oct 29 22:49:28.101417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.097550 5610 generic.go:356] "Generic (PLEG): container finished" podID="e92999ac-9100-4f7e-9010-f9f720d5e4a7" containerID="e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f" exitCode=0 Oct 29 22:49:28.101417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.098608 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" Oct 29 22:49:28.101417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.098972 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" event={"ID":"e92999ac-9100-4f7e-9010-f9f720d5e4a7","Type":"ContainerDied","Data":"e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f"} Oct 29 22:49:28.101417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.099022 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh" event={"ID":"e92999ac-9100-4f7e-9010-f9f720d5e4a7","Type":"ContainerDied","Data":"e95e57d806ba72805f0b52c800defdbf2cddab4973150503b915edba5dfaa432"} Oct 29 22:49:28.101417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.099051 5610 scope.go:122] "RemoveContainer" containerID="e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f" Oct 29 22:49:28.133632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.115942 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92999ac-9100-4f7e-9010-f9f720d5e4a7-kube-api-access-clnvs\" (UniqueName: \"kubernetes.io/projected/e92999ac-9100-4f7e-9010-f9f720d5e4a7-kube-api-access-clnvs\") pod \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\" (UID: \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\") " Oct 29 22:49:28.133632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.130144 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92999ac-9100-4f7e-9010-f9f720d5e4a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92999ac-9100-4f7e-9010-f9f720d5e4a7-plugins-config-volume\") pod \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\" (UID: \"e92999ac-9100-4f7e-9010-f9f720d5e4a7\") " Oct 29 22:49:28.139321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.135804 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92999ac-9100-4f7e-9010-f9f720d5e4a7-plugins-config-volume" pod "e92999ac-9100-4f7e-9010-f9f720d5e4a7" (UID: "e92999ac-9100-4f7e-9010-f9f720d5e4a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:49:28.139321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.138016 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92999ac-9100-4f7e-9010-f9f720d5e4a7-kube-api-access-clnvs" pod "e92999ac-9100-4f7e-9010-f9f720d5e4a7" (UID: "e92999ac-9100-4f7e-9010-f9f720d5e4a7"). InnerVolumeSpecName "kube-api-access-clnvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:28.149871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.149844 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:49:28.154179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.154164 5610 scope.go:122] "RemoveContainer" containerID="e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f" Oct 29 22:49:28.154636 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:49:28.154605 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f\": container with ID starting with e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f not found: ID does not exist" containerID="e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f" Oct 29 22:49:28.154715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.154649 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f"} err="failed to get container status \"e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f\": rpc error: code = NotFound desc = could not find container \"e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f\": container with ID starting with e57096bc78569f99bc447adf2b95a5cce7048647f36f6192d774cde915f6004f not found: ID does not exist" Oct 29 22:49:28.231987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.231913 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1530f5dc-2049-48f7-aef2-55d5e5956f5f-kube-api-access-bn79b\" (UniqueName: \"kubernetes.io/projected/1530f5dc-2049-48f7-aef2-55d5e5956f5f-kube-api-access-bn79b\") pod \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\" (UID: \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\") " Oct 29 22:49:28.232130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.231993 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1530f5dc-2049-48f7-aef2-55d5e5956f5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f5dc-2049-48f7-aef2-55d5e5956f5f-plugins-config-volume\") pod \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\" (UID: \"1530f5dc-2049-48f7-aef2-55d5e5956f5f\") " Oct 29 22:49:28.232653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.232626 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1530f5dc-2049-48f7-aef2-55d5e5956f5f-plugins-config-volume" pod "1530f5dc-2049-48f7-aef2-55d5e5956f5f" (UID: "1530f5dc-2049-48f7-aef2-55d5e5956f5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:49:28.233044 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.233025 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f5dc-2049-48f7-aef2-55d5e5956f5f-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:28.233105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.233048 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnvs\" (UniqueName: \"kubernetes.io/projected/e92999ac-9100-4f7e-9010-f9f720d5e4a7-kube-api-access-clnvs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:28.233105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.233057 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e92999ac-9100-4f7e-9010-f9f720d5e4a7-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:28.234039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.234018 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1530f5dc-2049-48f7-aef2-55d5e5956f5f-kube-api-access-bn79b" pod "1530f5dc-2049-48f7-aef2-55d5e5956f5f" (UID: "1530f5dc-2049-48f7-aef2-55d5e5956f5f"). InnerVolumeSpecName "kube-api-access-bn79b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:28.335030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.334993 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn79b\" (UniqueName: \"kubernetes.io/projected/1530f5dc-2049-48f7-aef2-55d5e5956f5f-kube-api-access-bn79b\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:28.356982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.356950 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:28.516256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.516160 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh"] Oct 29 22:49:28.525297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:28.525267 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-139eb7-746465db5b-7l9zh"] Oct 29 22:49:29.017884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.017845 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:49:29.018045 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.017901 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:49:29.110624 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.110595 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" Oct 29 22:49:29.111527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.110632 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg" event={"ID":"1530f5dc-2049-48f7-aef2-55d5e5956f5f","Type":"ContainerDied","Data":"50989d468b8b85dcf496447b30f8cf184ea1f9a7b707bed0d32c28f0e9b91346"} Oct 29 22:49:29.111527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.110704 5610 scope.go:122] "RemoveContainer" containerID="fd75b80eb24ff752a93de1a80bb8542190c904c2c3bb45828c34490d66995cfa" Oct 29 22:49:29.157972 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.157948 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg"] Oct 29 22:49:29.197441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.196682 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-139eb7-56594ffbcf-grqkg"] Oct 29 22:49:29.638692 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.638605 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1530f5dc-2049-48f7-aef2-55d5e5956f5f" path="/var/lib/kubelet/pods/1530f5dc-2049-48f7-aef2-55d5e5956f5f/volumes" Oct 29 22:49:29.639140 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.639123 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92999ac-9100-4f7e-9010-f9f720d5e4a7" path="/var/lib/kubelet/pods/e92999ac-9100-4f7e-9010-f9f720d5e4a7/volumes" Oct 29 22:49:29.715677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.715646 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.867967 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-certs\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.868039 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48702783-46a4-4dec-9dab-3c33fcb68587-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48702783-46a4-4dec-9dab-3c33fcb68587-istiod-ca-cert\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.868071 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-socket\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.868173 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48702783-46a4-4dec-9dab-3c33fcb68587-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48702783-46a4-4dec-9dab-3c33fcb68587-istio-podinfo\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.868230 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-data\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.868408 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-kube-api-access-96s8w\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-kube-api-access-96s8w\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.868463 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-credential-socket\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.868559 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-istio-token\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-istio-token\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.869758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.868616 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-envoy\") pod \"48702783-46a4-4dec-9dab-3c33fcb68587\" (UID: \"48702783-46a4-4dec-9dab-3c33fcb68587\") " Oct 29 22:49:29.889096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.888845 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-envoy" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.896937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.889302 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-data" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.896937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.896896 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-credential-socket" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.897694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.897475 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-certs" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.897811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.897722 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-socket" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:49:29.900418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.897881 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48702783-46a4-4dec-9dab-3c33fcb68587-istiod-ca-cert" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:49:29.900418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.899860 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48702783-46a4-4dec-9dab-3c33fcb68587-istio-podinfo" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:49:29.900418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.900196 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-istio-token" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:29.900418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.900311 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-kube-api-access-96s8w" pod "48702783-46a4-4dec-9dab-3c33fcb68587" (UID: "48702783-46a4-4dec-9dab-3c33fcb68587"). InnerVolumeSpecName "kube-api-access-96s8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979838 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96s8w\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-kube-api-access-96s8w\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979883 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979899 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48702783-46a4-4dec-9dab-3c33fcb68587-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979915 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979928 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979941 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48702783-46a4-4dec-9dab-3c33fcb68587-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979954 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979969 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48702783-46a4-4dec-9dab-3c33fcb68587-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:29.980009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:29.979982 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48702783-46a4-4dec-9dab-3c33fcb68587-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:49:30.017773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.017743 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:49:30.017943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.017865 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:49:30.123009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.122965 5610 generic.go:356] "Generic (PLEG): container finished" podID="48702783-46a4-4dec-9dab-3c33fcb68587" containerID="e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054" exitCode=0 Oct 29 22:49:30.123513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.123025 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" Oct 29 22:49:30.123513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.123048 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" event={"ID":"48702783-46a4-4dec-9dab-3c33fcb68587","Type":"ContainerDied","Data":"e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054"} Oct 29 22:49:30.123513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.123100 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd" event={"ID":"48702783-46a4-4dec-9dab-3c33fcb68587","Type":"ContainerDied","Data":"899b57920d005189f68f9e9e517ce60be3d136899e76250287711caa6dd69880"} Oct 29 22:49:30.123513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.123121 5610 scope.go:122] "RemoveContainer" containerID="e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054" Oct 29 22:49:30.135110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.135094 5610 scope.go:122] "RemoveContainer" containerID="e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054" Oct 29 22:49:30.135434 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:49:30.135411 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054\": container with ID starting with e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054 not found: ID does not exist" containerID="e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054" Oct 29 22:49:30.135522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.135440 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054"} err="failed to get container status \"e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054\": rpc error: code = NotFound desc = could not find container \"e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054\": container with ID starting with e2892c156e7ac2ec8e98a7b61d6ea6546fee21509021d990fbe96d0b12577054 not found: ID does not exist" Oct 29 22:49:30.182476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.181344 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:30.210419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.209446 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd"] Oct 29 22:49:30.245418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:30.241074 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-139eb7-openshift-default-77847fb8b7-ckcwd"] Oct 29 22:49:31.018196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:31.018157 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:49:31.018374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:31.018213 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:49:31.639581 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:31.639549 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" path="/var/lib/kubelet/pods/48702783-46a4-4dec-9dab-3c33fcb68587/volumes" Oct 29 22:49:32.029417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:32.021778 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 29 22:49:32.029417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:32.021837 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 29 22:49:32.237319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:32.237287 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:33.023549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:33.023520 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:33.024050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:33.024024 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:33.024704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:33.024680 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 22:49:33.535830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:33.535789 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:35.057409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:35.057358 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:36.607156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:36.607109 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:36.754451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:36.754327 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:37.548724 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:37.548681 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:38.251656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:38.251621 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:39.097542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:39.097503 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:39.261538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:39.261498 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:40.010446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:40.010409 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:41.616810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:41.616767 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:43.348791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:43.348744 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:43.500585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:43.500525 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:43.828466 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:49:43.827507 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:49:45.413472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:45.413163 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:45.687829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:45.687685 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:47.331049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:47.331012 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:49.041953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:49.041920 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:50.619764 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:50.619721 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:52.098212 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:52.098159 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:53.722044 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:53.721946 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:54.042492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:54.042404 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:54.191378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:54.191337 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:55.254415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:55.254305 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:56.803259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:56.803198 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:58.269686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:58.269643 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:49:59.678895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:49:59.678862 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:01.475991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.475948 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:01.912125 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.912040 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5"] Oct 29 22:50:01.913525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913494 5610 cpu_manager.go:414] "containerMap: removing container" podUID="e92999ac-9100-4f7e-9010-f9f720d5e4a7" containerName="payload-processing" Oct 29 22:50:01.913525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913520 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92999ac-9100-4f7e-9010-f9f720d5e4a7" containerName="payload-processing" Oct 29 22:50:01.913734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913557 5610 cpu_manager.go:414] "containerMap: removing container" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" Oct 29 22:50:01.913734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913565 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" Oct 29 22:50:01.913734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913593 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1530f5dc-2049-48f7-aef2-55d5e5956f5f" containerName="payload-pre-processing" Oct 29 22:50:01.913734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913600 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1530f5dc-2049-48f7-aef2-55d5e5956f5f" containerName="payload-pre-processing" Oct 29 22:50:01.913948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913788 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="48702783-46a4-4dec-9dab-3c33fcb68587" containerName="istio-proxy" Oct 29 22:50:01.913948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913809 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1530f5dc-2049-48f7-aef2-55d5e5956f5f" containerName="payload-pre-processing" Oct 29 22:50:01.913948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.913834 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92999ac-9100-4f7e-9010-f9f720d5e4a7" containerName="payload-processing" Oct 29 22:50:01.918294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.918272 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:01.937172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.937147 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-45e508\"/\"kube-root-ca.crt\"" Oct 29 22:50:01.957024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.956994 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-45e508\"/\"default-dockercfg-46dct\"" Oct 29 22:50:01.957253 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.957232 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-45e508\"/\"openshift-service-ca.crt\"" Oct 29 22:50:01.957491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.957468 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-45e508\"/\"test-model-45e508-kserve-self-signed-certs\"" Oct 29 22:50:01.963157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:01.963127 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5"] Oct 29 22:50:02.065677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.065638 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-home\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.065936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.065710 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2cqm\" (UniqueName: \"kubernetes.io/projected/4a99d1a9-82e9-4475-a37a-74f69314dad0-kube-api-access-d2cqm\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.065936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.065761 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-tmp-dir\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.065936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.065907 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a99d1a9-82e9-4475-a37a-74f69314dad0-tls-certs\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.066137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.065961 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-model-cache\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.066137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.066107 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-dshm\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.085267 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.085232 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:02.167349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.167230 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a99d1a9-82e9-4475-a37a-74f69314dad0-tls-certs\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.167349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.167307 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-model-cache\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.167675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.167421 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-dshm\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.167675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.167637 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-home\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.167812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.167693 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2cqm\" (UniqueName: \"kubernetes.io/projected/4a99d1a9-82e9-4475-a37a-74f69314dad0-kube-api-access-d2cqm\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.167812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.167740 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-tmp-dir\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.168431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.168363 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-model-cache\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.168704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.168673 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-home\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.168798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.168683 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-tmp-dir\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.170733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.170691 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a99d1a9-82e9-4475-a37a-74f69314dad0-tls-certs\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.171326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.171210 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-dshm\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.185594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.185570 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2cqm\" (UniqueName: \"kubernetes.io/projected/4a99d1a9-82e9-4475-a37a-74f69314dad0-kube-api-access-d2cqm\") pod \"test-model-45e508-kserve-c75c89dc7-t5tc5\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.249419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.248870 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:02.747434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:02.747409 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5"] Oct 29 22:50:02.749282 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:50:02.749253 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a99d1a9_82e9_4475_a37a_74f69314dad0.slice/crio-4e330ac579296ab743ea0c7d369b7174d9fb09ec18f034e4f03ae3dd86706e23 WatchSource:0}: Error finding container 4e330ac579296ab743ea0c7d369b7174d9fb09ec18f034e4f03ae3dd86706e23: Status 404 returned error can't find the container with id 4e330ac579296ab743ea0c7d369b7174d9fb09ec18f034e4f03ae3dd86706e23 Oct 29 22:50:03.119476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:03.119380 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:03.541337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:03.541300 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" event={"ID":"4a99d1a9-82e9-4475-a37a-74f69314dad0","Type":"ContainerStarted","Data":"2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2"} Oct 29 22:50:03.541530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:03.541351 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" event={"ID":"4a99d1a9-82e9-4475-a37a-74f69314dad0","Type":"ContainerStarted","Data":"4e330ac579296ab743ea0c7d369b7174d9fb09ec18f034e4f03ae3dd86706e23"} Oct 29 22:50:03.576804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:03.576759 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" podStartSLOduration=2.5767469 podStartE2EDuration="2.5767469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:03.575951143 +0000 UTC m=+2066.057268637" watchObservedRunningTime="2026-10-29 22:50:03.5767469 +0000 UTC m=+2066.058064437" Oct 29 22:50:04.249989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:04.249961 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:04.729938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:04.711501 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:06.332706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:06.332669 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:07.846385 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:07.846355 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:09.321826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:09.321788 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:11.061130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:11.061092 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:12.249804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:12.249767 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:12.263257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:12.263231 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:12.626636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:12.626557 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:50:12.657915 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:12.657864 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:13.713217 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:50:13.713182 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:50:14.168188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:14.168053 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:15.708491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:15.708449 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:17.324369 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:17.324320 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:19.000922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:19.000884 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:20.444679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:20.444587 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:21.413552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.413373 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8"] Oct 29 22:50:21.419873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.419847 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.438350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.438322 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8"] Oct 29 22:50:21.442693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.442664 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-57d336\"/\"kube-root-ca.crt\"" Oct 29 22:50:21.442837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.442713 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-57d336\"/\"default-dockercfg-tbsw8\"" Oct 29 22:50:21.442837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.442728 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-57d336\"/\"openshift-service-ca.crt\"" Oct 29 22:50:21.442837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.442744 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-57d336\"/\"test-model-57d336-kserve-self-signed-certs\"" Oct 29 22:50:21.502553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.502496 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:21.553449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.553405 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-dshm\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.553611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.553502 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tls-certs\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.553676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.553659 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqtw6\" (UniqueName: \"kubernetes.io/projected/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-kube-api-access-zqtw6\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.553719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.553698 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-model-cache\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.553830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.553818 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tmp-dir\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.553875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.553849 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-home\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.664461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.663584 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqtw6\" (UniqueName: \"kubernetes.io/projected/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-kube-api-access-zqtw6\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.664461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.663658 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-model-cache\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.664461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.663809 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tmp-dir\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.664461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.663855 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-home\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.664461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.663945 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-dshm\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.664461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.663995 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tls-certs\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.672411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.671116 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-model-cache\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.672411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.671386 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tmp-dir\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.672411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.671642 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-home\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.707826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.706585 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-dshm\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.727584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.713006 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tls-certs\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:21.766534 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:21.766446 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqtw6\" (UniqueName: \"kubernetes.io/projected/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-kube-api-access-zqtw6\") pod \"test-model-57d336-kserve-85b9c985f9-76sr8\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:22.042400 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:22.042372 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:22.168612 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:22.149955 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:22.362562 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:50:22.362528 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd19e93b_f11d_4cd5_b2ad_bb2740892b87.slice/crio-4aff39aa43509ff24ea590a9cec48a7d1dcadb8c63948bb9687ebff10d67cd4f WatchSource:0}: Error finding container 4aff39aa43509ff24ea590a9cec48a7d1dcadb8c63948bb9687ebff10d67cd4f: Status 404 returned error can't find the container with id 4aff39aa43509ff24ea590a9cec48a7d1dcadb8c63948bb9687ebff10d67cd4f Oct 29 22:50:22.363257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:22.363225 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8"] Oct 29 22:50:22.792603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:22.792560 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" event={"ID":"bd19e93b-f11d-4cd5-b2ad-bb2740892b87","Type":"ContainerStarted","Data":"6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a"} Oct 29 22:50:22.792603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:22.792620 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" event={"ID":"bd19e93b-f11d-4cd5-b2ad-bb2740892b87","Type":"ContainerStarted","Data":"4aff39aa43509ff24ea590a9cec48a7d1dcadb8c63948bb9687ebff10d67cd4f"} Oct 29 22:50:22.832426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:22.832349 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" podStartSLOduration=1.8323311420000001 podStartE2EDuration="1.832331142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:22.830244834 +0000 UTC m=+2085.311562328" watchObservedRunningTime="2026-10-29 22:50:22.832331142 +0000 UTC m=+2085.313648635" Oct 29 22:50:23.048470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:23.047589 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:23.651541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:23.651505 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:23.985471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:23.985027 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:25.714418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:25.713708 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:27.241210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:27.241178 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:28.926994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:28.926941 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:29.195534 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:29.195501 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:30.633311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:30.633257 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:32.042944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:32.042908 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:32.056716 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:32.056690 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:32.261511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:32.261476 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:33.176305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:33.176273 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:50:34.129416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:34.125899 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:35.777319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:35.777238 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:37.314601 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:37.314556 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:38.889801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:38.889752 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:39.215122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:39.215085 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:40.492417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:40.483114 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:41.956903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:41.956852 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:43.543897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:43.543844 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:43.711840 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:50:43.711808 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:50:45.276419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:45.275834 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:46.713437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:46.713388 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:48.217147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:48.217109 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:50.026435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:50.025521 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:51.696733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:51.696667 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:52.537076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.537039 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk"] Oct 29 22:50:52.542120 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.542099 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.557593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.557566 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-45e508-openshift-default-dockercfg-lpjww\"" Oct 29 22:50:52.570474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.570446 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk"] Oct 29 22:50:52.571738 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.571710 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-credential-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.571974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.571948 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wd8\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-kube-api-access-t4wd8\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.572172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.572143 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-envoy\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.572294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.572202 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-data\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.572294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.572277 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-podinfo\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.572470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.572321 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-token\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.572470 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.572347 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istiod-ca-cert\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.572591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.572550 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-certs\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.572591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.572586 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.673968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.673930 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-certs\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.674166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674093 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.674337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674312 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-credential-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.674439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674358 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wd8\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-kube-api-access-t4wd8\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.674661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674566 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-envoy\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.674661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674602 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.674661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674654 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-data\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.675794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674698 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-podinfo\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.675794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674732 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-token\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.675794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674310 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-certs\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.675794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.674781 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istiod-ca-cert\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.675794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.675118 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-credential-socket\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.675794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.675425 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-data\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.675794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.675577 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istiod-ca-cert\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.677310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.677279 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-envoy\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.677961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.677941 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-podinfo\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.692282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.692254 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-token\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.692493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.692471 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wd8\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-kube-api-access-t4wd8\") pod \"e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.858434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.858323 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:52.983573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:52.983505 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:53.126168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:53.126093 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:53.188933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:53.188902 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk"] Oct 29 22:50:53.189559 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:50:53.189534 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f82bd6f_638c_45f1_8bd0_2d7d402c881b.slice/crio-4504ea6bcea1a3ecb60be0f668786ee1ddd809bff3c94a74659c71087e87299d WatchSource:0}: Error finding container 4504ea6bcea1a3ecb60be0f668786ee1ddd809bff3c94a74659c71087e87299d: Status 404 returned error can't find the container with id 4504ea6bcea1a3ecb60be0f668786ee1ddd809bff3c94a74659c71087e87299d Oct 29 22:50:53.192907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:53.192873 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:50:53.193022 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:53.192960 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:50:53.193022 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:53.193007 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:50:53.276484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:53.276440 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:53.400841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:53.400805 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" event={"ID":"6f82bd6f-638c-45f1-8bd0-2d7d402c881b","Type":"ContainerStarted","Data":"4504ea6bcea1a3ecb60be0f668786ee1ddd809bff3c94a74659c71087e87299d"} Oct 29 22:50:54.018446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.018410 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8"] Oct 29 22:50:54.024135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.024112 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.041997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.041972 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-172c10-openshift-default-dockercfg-sbfcd\"" Oct 29 22:50:54.058464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.058437 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8"] Oct 29 22:50:54.105119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105087 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-istio-token\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.105297 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105129 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-certs\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.105405 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105368 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f3562d-4be8-436f-8435-f972c3e98761-istio-podinfo\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.105541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105525 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-envoy\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.105603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105557 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-data\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.105603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105590 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwwxc\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-kube-api-access-fwwxc\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.105717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105668 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-credential-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.105717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105687 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f3562d-4be8-436f-8435-f972c3e98761-istiod-ca-cert\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.105842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.105737 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.207814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.207776 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f3562d-4be8-436f-8435-f972c3e98761-istio-podinfo\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.208020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.207921 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-envoy\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.208020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.207960 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-data\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.208020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.208003 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwwxc\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-kube-api-access-fwwxc\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.208363 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.208340 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-credential-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.208502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.208476 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-data\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.208719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.208696 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-credential-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.208800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.208776 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f3562d-4be8-436f-8435-f972c3e98761-istiod-ca-cert\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.208868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.208817 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.209261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.209237 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-socket\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.209367 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.209348 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-istio-token\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.209463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.209448 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-certs\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.209648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.209623 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f3562d-4be8-436f-8435-f972c3e98761-istiod-ca-cert\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.209788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.209761 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-certs\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.210975 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.210942 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-envoy\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.211370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.211340 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f3562d-4be8-436f-8435-f972c3e98761-istio-podinfo\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.225231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.225203 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwwxc\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-kube-api-access-fwwxc\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.225952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.225931 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-istio-token\") pod \"e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.341490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.341403 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:54.435488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.425140 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" event={"ID":"6f82bd6f-638c-45f1-8bd0-2d7d402c881b","Type":"ContainerStarted","Data":"4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4"} Oct 29 22:50:54.499568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.499507 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" podStartSLOduration=2.499487073 podStartE2EDuration="2.499487073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:54.499350498 +0000 UTC m=+2116.980668008" watchObservedRunningTime="2026-10-29 22:50:54.499487073 +0000 UTC m=+2116.980804566" Oct 29 22:50:54.710964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.710925 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8"] Oct 29 22:50:54.712932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.712902 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:50:54.713018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.712969 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:50:54.713018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.712999 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:50:54.858631 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.858599 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:50:54.868416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.867515 5610 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 29 22:50:54.868416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:54.867571 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 29 22:50:55.203243 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:55.203076 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:55.435310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:55.435272 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" event={"ID":"90f3562d-4be8-436f-8435-f972c3e98761","Type":"ContainerStarted","Data":"0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7"} Oct 29 22:50:55.435545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:55.435376 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" event={"ID":"90f3562d-4be8-436f-8435-f972c3e98761","Type":"ContainerStarted","Data":"3835510dc58938848466783fde842f3a39b4d875b39c542d19a12578a48c4a23"} Oct 29 22:50:55.477535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:55.477420 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podStartSLOduration=2.477383218 podStartE2EDuration="2.477383218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:50:55.470859443 +0000 UTC m=+2117.952176936" watchObservedRunningTime="2026-10-29 22:50:55.477383218 +0000 UTC m=+2117.958700712" Oct 29 22:50:55.859107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:55.858617 5610 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 29 22:50:55.859107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:55.858677 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 29 22:50:56.343037 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:56.342422 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:50:56.344206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:56.344181 5610 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:50:56.344286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:56.344228 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:50:56.858889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:56.858855 5610 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 29 22:50:56.859152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:56.859126 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 29 22:50:56.887506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:56.887374 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:57.342110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:57.342073 5610 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:50:57.342285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:57.342157 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:50:57.864425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:57.859584 5610 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 29 22:50:57.864425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:57.859664 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 29 22:50:58.342144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:58.342114 5610 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:50:58.342333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:58.342160 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:50:58.859238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:58.859201 5610 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 29 22:50:58.859448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:58.859248 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 29 22:50:58.895416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:58.891407 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:50:59.348351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:59.341882 5610 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:50:59.348351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:59.341962 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:50:59.859216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:59.859183 5610 patch_prober.go:29] interesting pod/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 29 22:50:59.859377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:50:59.859251 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 29 22:51:00.341811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:00.341774 5610 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:51:00.342216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:00.341824 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:51:00.877417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:00.876692 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:51:00.881416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:00.878378 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:51:00.883425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:00.882598 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:51:00.942296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:00.937672 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:01.342219 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:01.342179 5610 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 29 22:51:01.342683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:01.342257 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 29 22:51:02.246328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:02.246286 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:02.393235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:02.393212 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:51:02.396018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:02.395992 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:51:02.398817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:02.398798 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:51:02.726642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:02.726612 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:04.340668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:04.340631 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:05.987990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:05.987943 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:07.585503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:07.565797 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:09.086539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:09.086487 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:11.236427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:11.232497 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:13.046657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:13.046609 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:13.713069 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:51:13.713033 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:51:14.652184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:14.652144 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:16.300810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:16.300771 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:17.971970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:17.971926 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:19.618326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:19.618288 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:21.227885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:21.227807 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:21.334089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:21.326208 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q"] Oct 29 22:51:21.336104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:21.336064 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" containerID="cri-o://d2eea1347eb49afaa4a201b940cd8b2c34129f4502956ddfb63131d725e9204d" gracePeriod=30 Oct 29 22:51:22.347207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:22.347172 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr"] Oct 29 22:51:22.348071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:22.348034 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" containerID="cri-o://39656155f4fd52a1c2dc37b239593b31bacc3517449a71619541d94f55924d50" gracePeriod=30 Oct 29 22:51:22.876894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:22.876855 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:23.196473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:23.192788 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:24.447424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:24.446072 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:25.615996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:25.615951 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:26.321514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:26.313071 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:26.737409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:26.737370 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 29 22:51:26.737794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:26.737478 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 29 22:51:26.916612 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:26.916581 5610 generic.go:356] "Generic (PLEG): container finished" podID="230c618f-fa51-489b-9075-45cb39b854a3" containerID="d2eea1347eb49afaa4a201b940cd8b2c34129f4502956ddfb63131d725e9204d" exitCode=0 Oct 29 22:51:26.916794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:26.916659 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" event={"ID":"230c618f-fa51-489b-9075-45cb39b854a3","Type":"ContainerDied","Data":"d2eea1347eb49afaa4a201b940cd8b2c34129f4502956ddfb63131d725e9204d"} Oct 29 22:51:27.098574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.098172 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:51:27.223872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.223839 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-credential-socket\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.223884 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/230c618f-fa51-489b-9075-45cb39b854a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230c618f-fa51-489b-9075-45cb39b854a3-istio-podinfo\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.223922 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-kube-api-access-n5q79\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-kube-api-access-n5q79\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224015 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-envoy\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224220 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224070 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-data\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224220 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224153 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-socket\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224220 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224167 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-credential-socket" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.224220 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224205 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230c618f-fa51-489b-9075-45cb39b854a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230c618f-fa51-489b-9075-45cb39b854a3-istiod-ca-cert\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224386 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-istio-token\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-istio-token\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224437 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-data" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.224578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224480 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-socket" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.224578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224494 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-certs\") pod \"230c618f-fa51-489b-9075-45cb39b854a3\" (UID: \"230c618f-fa51-489b-9075-45cb39b854a3\") " Oct 29 22:51:27.224722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224678 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230c618f-fa51-489b-9075-45cb39b854a3-istiod-ca-cert" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:27.224831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.224810 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-certs" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.225325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.225304 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.225473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.225327 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.225473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.225340 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.225473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.225356 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/230c618f-fa51-489b-9075-45cb39b854a3-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.225473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.225369 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.226626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.226603 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-envoy" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:27.226736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.226679 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/230c618f-fa51-489b-9075-45cb39b854a3-istio-podinfo" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:51:27.227020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.226999 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-kube-api-access-n5q79" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "kube-api-access-n5q79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:27.227117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.227026 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-istio-token" pod "230c618f-fa51-489b-9075-45cb39b854a3" (UID: "230c618f-fa51-489b-9075-45cb39b854a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:27.327380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.327300 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.327380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.327326 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/230c618f-fa51-489b-9075-45cb39b854a3-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.327380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.327337 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5q79\" (UniqueName: \"kubernetes.io/projected/230c618f-fa51-489b-9075-45cb39b854a3-kube-api-access-n5q79\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.327380 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.327352 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/230c618f-fa51-489b-9075-45cb39b854a3-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:27.929107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.928965 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" Oct 29 22:51:27.929107 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.928968 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q" event={"ID":"230c618f-fa51-489b-9075-45cb39b854a3","Type":"ContainerDied","Data":"d07c6bc182c4028e5df7e5cf0d4d285cd1179f46f6c9fd7b9bab2181c87f0d0c"} Oct 29 22:51:27.929830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.929154 5610 scope.go:122] "RemoveContainer" containerID="d2eea1347eb49afaa4a201b940cd8b2c34129f4502956ddfb63131d725e9204d" Oct 29 22:51:27.932747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.932722 5610 generic.go:356] "Generic (PLEG): container finished" podID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerID="39656155f4fd52a1c2dc37b239593b31bacc3517449a71619541d94f55924d50" exitCode=0 Oct 29 22:51:27.932886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.932759 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" event={"ID":"295069a5-7550-4e03-b1aa-f6c7ae3aa310","Type":"ContainerDied","Data":"39656155f4fd52a1c2dc37b239593b31bacc3517449a71619541d94f55924d50"} Oct 29 22:51:27.949758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.949729 5610 scope.go:122] "RemoveContainer" containerID="43a669058423f61f015957066ca288335db15e39653dec0a834d736fba87aa69" Oct 29 22:51:27.993514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:27.992176 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z"] Oct 29 22:51:28.018437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.017960 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q"] Oct 29 22:51:28.020140 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.020108 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-2nk7q"] Oct 29 22:51:28.045176 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.045146 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:28.214992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.213087 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:28.261417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.253846 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.265673 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-kube-api-access-gs7cx\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-kube-api-access-gs7cx\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.265821 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-envoy\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.265851 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-token\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-token\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.265932 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-podinfo\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.266030 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-data\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.266156 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-certs\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.266181 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istiod-ca-cert\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.266222 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-socket\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.266283 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-credential-socket\") pod \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\" (UID: \"295069a5-7550-4e03-b1aa-f6c7ae3aa310\") " Oct 29 22:51:28.267293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.267210 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-credential-socket" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:28.268379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.268346 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-certs" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:28.269239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.268646 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istiod-ca-cert" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:51:28.269239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.269031 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-socket" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:28.269239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.269095 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-data" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:28.280485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.274887 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-token" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:28.280485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.275022 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-kube-api-access-gs7cx" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "kube-api-access-gs7cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:28.280485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.276495 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-envoy" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:28.280485 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.276500 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-podinfo" pod "295069a5-7550-4e03-b1aa-f6c7ae3aa310" (UID: "295069a5-7550-4e03-b1aa-f6c7ae3aa310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:51:28.368997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.368965 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.368997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.368994 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.368997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.369003 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.369319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.369011 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.369319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.369020 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.369319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.369039 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295069a5-7550-4e03-b1aa-f6c7ae3aa310-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.369319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.369047 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.369319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.369065 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295069a5-7550-4e03-b1aa-f6c7ae3aa310-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.369319 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.369076 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7cx\" (UniqueName: \"kubernetes.io/projected/295069a5-7550-4e03-b1aa-f6c7ae3aa310-kube-api-access-gs7cx\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:28.942124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.942098 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" Oct 29 22:51:28.942620 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.942137 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr" event={"ID":"295069a5-7550-4e03-b1aa-f6c7ae3aa310","Type":"ContainerDied","Data":"7e994546a8fa9c5093f670ab08932c81a23b4468c0fbd434419c5449e123e552"} Oct 29 22:51:28.942620 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.942202 5610 scope.go:122] "RemoveContainer" containerID="39656155f4fd52a1c2dc37b239593b31bacc3517449a71619541d94f55924d50" Oct 29 22:51:28.990052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.990017 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr"] Oct 29 22:51:28.999236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:28.999210 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-95672c-openshift-default-5774db9bc-jsggr"] Oct 29 22:51:29.639316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:29.639282 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230c618f-fa51-489b-9075-45cb39b854a3" path="/var/lib/kubelet/pods/230c618f-fa51-489b-9075-45cb39b854a3/volumes" Oct 29 22:51:29.639910 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:29.639894 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" path="/var/lib/kubelet/pods/295069a5-7550-4e03-b1aa-f6c7ae3aa310/volumes" Oct 29 22:51:29.665750 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:29.665707 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:31.424681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:31.424644 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:33.063417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:33.058676 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:34.539425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:34.536522 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:36.327771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.327486 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:36.442549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.437209 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v"] Oct 29 22:51:36.442549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.438486 5610 cpu_manager.go:414] "containerMap: removing container" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" Oct 29 22:51:36.442549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.438503 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" Oct 29 22:51:36.442549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.438566 5610 cpu_manager.go:414] "containerMap: removing container" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" Oct 29 22:51:36.442549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.438574 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" Oct 29 22:51:36.442549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.439997 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" Oct 29 22:51:36.442549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.440022 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="230c618f-fa51-489b-9075-45cb39b854a3" containerName="istio-proxy" Oct 29 22:51:36.449483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.445735 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.449483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.446364 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v"] Oct 29 22:51:36.463219 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.463191 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-6bb06a-openshift-default-dockercfg-vhrww\"" Oct 29 22:51:36.548487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.548449 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.548666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.548617 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqcr\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-kube-api-access-5kqcr\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.548732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.548673 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.548794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.548779 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.548906 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.548891 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.548960 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.548927 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.549084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.549060 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.549149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.549134 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.549260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.549240 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.650831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.650748 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.650991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.650859 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.650991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.650891 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.650991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.650953 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.651124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.651017 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.651124 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.651099 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.651210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.651197 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.651260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.651223 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-credential-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.651324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.651297 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqcr\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-kube-api-access-5kqcr\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.651379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.651333 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.651499 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.651485 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-certs\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.651878 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.651783 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-socket\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.652032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.652008 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istiod-ca-cert\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.652115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.652042 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-data\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.654317 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.654289 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-podinfo\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.654458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.654367 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-envoy\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.668386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.668359 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqcr\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-kube-api-access-5kqcr\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.668649 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.668631 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-token\") pod \"e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:36.766491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:36.766461 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:37.091500 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:51:37.091463 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f49e776_3db8_439c_9b4f_fa685c3b8fa3.slice/crio-3f1c9c22c19559a1a67ea57e8150856465c760b4e7951e396a58858daf9d0c29 WatchSource:0}: Error finding container 3f1c9c22c19559a1a67ea57e8150856465c760b4e7951e396a58858daf9d0c29: Status 404 returned error can't find the container with id 3f1c9c22c19559a1a67ea57e8150856465c760b4e7951e396a58858daf9d0c29 Oct 29 22:51:37.091615 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:37.091546 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v"] Oct 29 22:51:37.094432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:37.094378 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:37.094517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:37.094470 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:37.094517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:37.094501 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:51:38.014428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:38.013931 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:38.168414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:38.162466 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" event={"ID":"8f49e776-3db8-439c-9b4f-fa685c3b8fa3","Type":"ContainerStarted","Data":"8a17ba0e840e65dc1edb3893359d8cef900b35c8334efdbf29b72390e68b496c"} Oct 29 22:51:38.168414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:38.162539 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" event={"ID":"8f49e776-3db8-439c-9b4f-fa685c3b8fa3","Type":"ContainerStarted","Data":"3f1c9c22c19559a1a67ea57e8150856465c760b4e7951e396a58858daf9d0c29"} Oct 29 22:51:38.234414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:38.234344 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" podStartSLOduration=2.234325376 podStartE2EDuration="2.234325376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:38.233873907 +0000 UTC m=+2160.715191403" watchObservedRunningTime="2026-10-29 22:51:38.234325376 +0000 UTC m=+2160.715642869" Oct 29 22:51:38.793418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:38.779758 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:38.793418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:38.787498 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:51:38.793418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:38.787568 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 29 22:51:39.324657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:39.324617 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:39.679407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:39.679292 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:39.767983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:39.767937 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:51:39.768555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:39.768493 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 29 22:51:40.778428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:40.778104 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:51:40.778428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:40.778173 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 29 22:51:41.568418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:41.567022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:41.778417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:41.777872 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:51:41.778417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:41.777959 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 29 22:51:42.769754 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:42.769716 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 29 22:51:42.792414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:42.770018 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 29 22:51:43.045824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.045725 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" podUID="ecb5ed71-07f9-4334-aab8-cef607b23d5d" containerName="main" containerID="cri-o://c7b3946f801b8efe2b0c15e13f9496312e22c976ac7b180aff88580316d67c4c" gracePeriod=45 Oct 29 22:51:43.247776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.247747 5610 generic.go:356] "Generic (PLEG): container finished" podID="ecb5ed71-07f9-4334-aab8-cef607b23d5d" containerID="c7b3946f801b8efe2b0c15e13f9496312e22c976ac7b180aff88580316d67c4c" exitCode=0 Oct 29 22:51:43.248010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.247817 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" event={"ID":"ecb5ed71-07f9-4334-aab8-cef607b23d5d","Type":"ContainerDied","Data":"c7b3946f801b8efe2b0c15e13f9496312e22c976ac7b180aff88580316d67c4c"} Oct 29 22:51:43.313988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.308481 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:43.476503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.476466 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:51:43.558546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.558506 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb5ed71-07f9-4334-aab8-cef607b23d5d-kube-api-access-8h8cb\" (UniqueName: \"kubernetes.io/projected/ecb5ed71-07f9-4334-aab8-cef607b23d5d-kube-api-access-8h8cb\") pod \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " Oct 29 22:51:43.558722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.558629 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-dshm\") pod \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " Oct 29 22:51:43.558722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.558695 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tmp-dir\") pod \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " Oct 29 22:51:43.558828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.558744 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tls-certs\") pod \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " Oct 29 22:51:43.558891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.558848 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-home\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-home\") pod \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " Oct 29 22:51:43.559323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.559029 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tmp-dir" pod "ecb5ed71-07f9-4334-aab8-cef607b23d5d" (UID: "ecb5ed71-07f9-4334-aab8-cef607b23d5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:43.559323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.559064 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-model-cache\") pod \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\" (UID: \"ecb5ed71-07f9-4334-aab8-cef607b23d5d\") " Oct 29 22:51:43.559323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.559264 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-home" pod "ecb5ed71-07f9-4334-aab8-cef607b23d5d" (UID: "ecb5ed71-07f9-4334-aab8-cef607b23d5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:43.559323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.559287 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-model-cache" pod "ecb5ed71-07f9-4334-aab8-cef607b23d5d" (UID: "ecb5ed71-07f9-4334-aab8-cef607b23d5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:43.559885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.559857 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:43.559970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.559915 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:43.559970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.559928 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:43.561773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.561736 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tls-certs" pod "ecb5ed71-07f9-4334-aab8-cef607b23d5d" (UID: "ecb5ed71-07f9-4334-aab8-cef607b23d5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:51:43.561773 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.561754 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-dshm" pod "ecb5ed71-07f9-4334-aab8-cef607b23d5d" (UID: "ecb5ed71-07f9-4334-aab8-cef607b23d5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:51:43.562081 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.562057 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb5ed71-07f9-4334-aab8-cef607b23d5d-kube-api-access-8h8cb" pod "ecb5ed71-07f9-4334-aab8-cef607b23d5d" (UID: "ecb5ed71-07f9-4334-aab8-cef607b23d5d"). InnerVolumeSpecName "kube-api-access-8h8cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:51:43.661670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.661638 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h8cb\" (UniqueName: \"kubernetes.io/projected/ecb5ed71-07f9-4334-aab8-cef607b23d5d-kube-api-access-8h8cb\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:43.661670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.661670 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb5ed71-07f9-4334-aab8-cef607b23d5d-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:43.661903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.661689 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb5ed71-07f9-4334-aab8-cef607b23d5d-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:51:43.726946 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:51:43.726906 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:51:43.836909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.836809 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:43.837507 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.837455 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:43.838572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:43.838552 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 22:51:44.258351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:44.258322 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" Oct 29 22:51:44.258351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:44.258346 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" event={"ID":"ecb5ed71-07f9-4334-aab8-cef607b23d5d","Type":"ContainerDied","Data":"ce06a48fa53a10c475aa51eded1c5d3b119ea8a09294c21b41072f111acd8ad1"} Oct 29 22:51:44.258626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:44.258434 5610 scope.go:122] "RemoveContainer" containerID="c7b3946f801b8efe2b0c15e13f9496312e22c976ac7b180aff88580316d67c4c" Oct 29 22:51:44.307354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:44.307318 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z"] Oct 29 22:51:44.311682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:44.311661 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z"] Oct 29 22:51:44.356744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:44.356702 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-379a6d/test-model-379a6d-kserve-df64688c-qfr6z" podUID="ecb5ed71-07f9-4334-aab8-cef607b23d5d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:51:44.829151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:44.829047 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:45.639918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:45.639884 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb5ed71-07f9-4334-aab8-cef607b23d5d" path="/var/lib/kubelet/pods/ecb5ed71-07f9-4334-aab8-cef607b23d5d/volumes" Oct 29 22:51:46.626450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:46.625090 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:47.755134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:47.755100 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj"] Oct 29 22:51:47.843274 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:47.843232 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:48.463530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:48.463440 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:50.024174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:50.024062 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:51.721112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:51.721053 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:52.381696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:52.381658 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:53.357660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:53.357626 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:54.814972 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:54.814905 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:56.543126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:56.543082 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:58.145104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:58.145063 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:51:59.727653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:51:59.727621 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:00.142546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:00.142452 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:01.677951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:01.677893 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:02.406607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:02.406573 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2"] Oct 29 22:52:02.495633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:02.495598 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:02.798883 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:02.798780 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" podUID="089f646b-d23c-481a-b1e3-19fce519cd91" containerName="main" containerID="cri-o://c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727" gracePeriod=45 Oct 29 22:52:03.145500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.122749 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:03.331231 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.331203 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:52:03.435814 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.435738 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/089f646b-d23c-481a-b1e3-19fce519cd91-tls-certs\" (UniqueName: \"kubernetes.io/secret/089f646b-d23c-481a-b1e3-19fce519cd91-tls-certs\") pod \"089f646b-d23c-481a-b1e3-19fce519cd91\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " Oct 29 22:52:03.435979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.435915 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-tmp-dir\") pod \"089f646b-d23c-481a-b1e3-19fce519cd91\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " Oct 29 22:52:03.436050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436026 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-home\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-home\") pod \"089f646b-d23c-481a-b1e3-19fce519cd91\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " Oct 29 22:52:03.436110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436070 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-model-cache\") pod \"089f646b-d23c-481a-b1e3-19fce519cd91\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " Oct 29 22:52:03.436187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436152 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-tmp-dir" pod "089f646b-d23c-481a-b1e3-19fce519cd91" (UID: "089f646b-d23c-481a-b1e3-19fce519cd91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:03.436187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436164 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089f646b-d23c-481a-b1e3-19fce519cd91-kube-api-access-f6wlp\" (UniqueName: \"kubernetes.io/projected/089f646b-d23c-481a-b1e3-19fce519cd91-kube-api-access-f6wlp\") pod \"089f646b-d23c-481a-b1e3-19fce519cd91\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " Oct 29 22:52:03.436354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436267 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-home" pod "089f646b-d23c-481a-b1e3-19fce519cd91" (UID: "089f646b-d23c-481a-b1e3-19fce519cd91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:03.436354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436298 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-dshm\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-dshm\") pod \"089f646b-d23c-481a-b1e3-19fce519cd91\" (UID: \"089f646b-d23c-481a-b1e3-19fce519cd91\") " Oct 29 22:52:03.436514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436382 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-model-cache" pod "089f646b-d23c-481a-b1e3-19fce519cd91" (UID: "089f646b-d23c-481a-b1e3-19fce519cd91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:03.436951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436930 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:03.437057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436952 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:03.437057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.436965 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:03.438084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.438064 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089f646b-d23c-481a-b1e3-19fce519cd91-tls-certs" pod "089f646b-d23c-481a-b1e3-19fce519cd91" (UID: "089f646b-d23c-481a-b1e3-19fce519cd91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:03.438286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.438262 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-dshm" pod "089f646b-d23c-481a-b1e3-19fce519cd91" (UID: "089f646b-d23c-481a-b1e3-19fce519cd91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:03.438387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.438319 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089f646b-d23c-481a-b1e3-19fce519cd91-kube-api-access-f6wlp" pod "089f646b-d23c-481a-b1e3-19fce519cd91" (UID: "089f646b-d23c-481a-b1e3-19fce519cd91"). InnerVolumeSpecName "kube-api-access-f6wlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:03.459752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.459721 5610 generic.go:356] "Generic (PLEG): container finished" podID="089f646b-d23c-481a-b1e3-19fce519cd91" containerID="c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727" exitCode=0 Oct 29 22:52:03.459881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.459783 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" Oct 29 22:52:03.459881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.459805 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" event={"ID":"089f646b-d23c-481a-b1e3-19fce519cd91","Type":"ContainerDied","Data":"c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727"} Oct 29 22:52:03.459881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.459859 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" event={"ID":"089f646b-d23c-481a-b1e3-19fce519cd91","Type":"ContainerDied","Data":"5877e29c5c6f4b3b67838334afe2f86df6a271971a043da106d4b4dd2d19764d"} Oct 29 22:52:03.460026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.459885 5610 scope.go:122] "RemoveContainer" containerID="c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727" Oct 29 22:52:03.473257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.473237 5610 scope.go:122] "RemoveContainer" containerID="c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727" Oct 29 22:52:03.473590 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:03.473571 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727\": container with ID starting with c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727 not found: ID does not exist" containerID="c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727" Oct 29 22:52:03.473696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.473601 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727"} err="failed to get container status \"c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727\": rpc error: code = NotFound desc = could not find container \"c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727\": container with ID starting with c4e0955f3f95cbadc8a8532bac544035358b0f9f65e44fbd26e1712ca4527727 not found: ID does not exist" Oct 29 22:52:03.508178 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.508153 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj"] Oct 29 22:52:03.512847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.512824 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj"] Oct 29 22:52:03.538747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.538723 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6wlp\" (UniqueName: \"kubernetes.io/projected/089f646b-d23c-481a-b1e3-19fce519cd91-kube-api-access-f6wlp\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:03.538747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.538745 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089f646b-d23c-481a-b1e3-19fce519cd91-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:03.538885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.538756 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089f646b-d23c-481a-b1e3-19fce519cd91-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:03.638073 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.638040 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089f646b-d23c-481a-b1e3-19fce519cd91" path="/var/lib/kubelet/pods/089f646b-d23c-481a-b1e3-19fce519cd91/volumes" Oct 29 22:52:03.696486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:03.693252 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:04.080123 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:04.080039 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6bc7b2/test-model-6bc7b2-kserve-769bd6fd79-r5kbj" podUID="089f646b-d23c-481a-b1e3-19fce519cd91" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:52:05.806419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:05.802456 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:06.987416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:06.985481 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" podUID="0e99b243-8803-4980-ab78-cc9c9a1ff7d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.187:8000/ready\": context deadline exceeded" Oct 29 22:52:07.007554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:07.007505 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:52:07.044169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:07.044124 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" podUID="bc290c4e-3d45-4547-bedc-ef18c50947b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.183:8000/ready\": context deadline exceeded" Oct 29 22:52:07.074573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:07.074494 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:52:07.135416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:07.133495 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" podUID="7ad9976c-7a0c-468b-b3f3-16fc3a567eda" containerName="main" probeResult="failure" output="Get \"https://10.134.0.185:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:52:07.232501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:07.232455 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 29 22:52:08.701676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:08.701592 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:10.243496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:10.243459 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:11.830416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:11.823584 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:13.214383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:13.214343 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:13.408233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:13.408195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:13.584182 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:13.584093 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:13.961666 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:13.961623 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:52:13.975334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:13.975241 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:14.146680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:14.146636 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:15.321472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:15.321429 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:16.278821 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:16.278781 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9"] Oct 29 22:52:16.279060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:16.279032 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" containerID="cri-o://3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9" gracePeriod=30 Oct 29 22:52:16.287022 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:16.286993 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7"] Oct 29 22:52:16.287240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:16.287193 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" containerID="cri-o://3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31" gracePeriod=30 Oct 29 22:52:16.294131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:16.294100 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr"] Oct 29 22:52:16.294327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:16.294305 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" containerID="cri-o://fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2" gracePeriod=30 Oct 29 22:52:17.055626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.055551 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:17.438412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.438315 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" podUID="7ad9976c-7a0c-468b-b3f3-16fc3a567eda" containerName="main" containerID="cri-o://0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1" gracePeriod=45 Oct 29 22:52:17.754458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.754431 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:52:17.909260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909228 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-model-cache\") pod \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " Oct 29 22:52:17.909442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909322 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-dshm\") pod \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " Oct 29 22:52:17.909442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909410 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tls-certs\") pod \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " Oct 29 22:52:17.909540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909454 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-home\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-home\") pod \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " Oct 29 22:52:17.909593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909550 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-kube-api-access-tfvms\" (UniqueName: \"kubernetes.io/projected/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-kube-api-access-tfvms\") pod \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " Oct 29 22:52:17.909593 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909557 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-model-cache" pod "7ad9976c-7a0c-468b-b3f3-16fc3a567eda" (UID: "7ad9976c-7a0c-468b-b3f3-16fc3a567eda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:17.909766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909742 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tmp-dir\") pod \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\" (UID: \"7ad9976c-7a0c-468b-b3f3-16fc3a567eda\") " Oct 29 22:52:17.909911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909771 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-home" pod "7ad9976c-7a0c-468b-b3f3-16fc3a567eda" (UID: "7ad9976c-7a0c-468b-b3f3-16fc3a567eda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:17.909982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.909926 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tmp-dir" pod "7ad9976c-7a0c-468b-b3f3-16fc3a567eda" (UID: "7ad9976c-7a0c-468b-b3f3-16fc3a567eda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:17.910440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.910420 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:17.910542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.910441 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:17.910542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.910453 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:17.911957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.911934 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-dshm" pod "7ad9976c-7a0c-468b-b3f3-16fc3a567eda" (UID: "7ad9976c-7a0c-468b-b3f3-16fc3a567eda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:17.912148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.912121 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tls-certs" pod "7ad9976c-7a0c-468b-b3f3-16fc3a567eda" (UID: "7ad9976c-7a0c-468b-b3f3-16fc3a567eda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:17.912224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.912169 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-kube-api-access-tfvms" pod "7ad9976c-7a0c-468b-b3f3-16fc3a567eda" (UID: "7ad9976c-7a0c-468b-b3f3-16fc3a567eda"). InnerVolumeSpecName "kube-api-access-tfvms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:17.975166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.975089 5610 generic.go:356] "Generic (PLEG): container finished" podID="7ad9976c-7a0c-468b-b3f3-16fc3a567eda" containerID="0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1" exitCode=0 Oct 29 22:52:17.975385 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.975362 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" Oct 29 22:52:17.975590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.975568 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" event={"ID":"7ad9976c-7a0c-468b-b3f3-16fc3a567eda","Type":"ContainerDied","Data":"0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1"} Oct 29 22:52:17.975713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.975697 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2" event={"ID":"7ad9976c-7a0c-468b-b3f3-16fc3a567eda","Type":"ContainerDied","Data":"38d9bddbe1f81e28d19d2442b72f40567776bc56e0d1570f879bf892f8d95109"} Oct 29 22:52:17.975840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.975826 5610 scope.go:122] "RemoveContainer" containerID="0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1" Oct 29 22:52:17.992669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.992647 5610 scope.go:122] "RemoveContainer" containerID="0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1" Oct 29 22:52:17.992973 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:17.992954 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1\": container with ID starting with 0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1 not found: ID does not exist" containerID="0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1" Oct 29 22:52:17.993039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:17.992980 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1"} err="failed to get container status \"0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1\": rpc error: code = NotFound desc = could not find container \"0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1\": container with ID starting with 0006ecabc24abea434b6421598ed34c2e246bb0ef0fcfed7ed1cb02a7b2527a1 not found: ID does not exist" Oct 29 22:52:18.012640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:18.012604 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.012757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:18.012658 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.012757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:18.012676 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfvms\" (UniqueName: \"kubernetes.io/projected/7ad9976c-7a0c-468b-b3f3-16fc3a567eda-kube-api-access-tfvms\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:18.024228 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:18.024203 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2"] Oct 29 22:52:18.030515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:18.030495 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8dd4c8/test-model-8dd4c8-kserve-6f69c5c4d5-sr7r2"] Oct 29 22:52:18.641031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:18.640991 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:19.638898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:19.638859 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad9976c-7a0c-468b-b3f3-16fc3a567eda" path="/var/lib/kubelet/pods/7ad9976c-7a0c-468b-b3f3-16fc3a567eda/volumes" Oct 29 22:52:20.231067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:20.231028 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:21.127052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.127012 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:21.127239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.127070 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:21.198109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.197676 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:21.198109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.197733 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:21.434120 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.433068 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng"] Oct 29 22:52:21.434839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.434666 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" podUID="61f84d72-b1ac-49c9-90d6-9a4735e61e4e" containerName="maas-api" containerID="cri-o://6b3081b52f50bb7a462ff821655a0120c72c2633cfc21434d8c8eca848172fa8" gracePeriod=30 Oct 29 22:52:21.530387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.530231 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:21.797755 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.776110 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:52:21.900650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.900238 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:52:21.930772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.930686 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:52:21.933077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.932670 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-token\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-token\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.933077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.932736 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-socket\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.933077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.932852 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-data\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933000 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-envoy\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933436 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-certs\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933083 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-socket" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933239 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-data" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933530 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-podinfo\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933597 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istiod-ca-cert\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933633 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-kube-api-access-vlmsj\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-kube-api-access-vlmsj\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933699 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-credential-socket\") pod \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\" (UID: \"a0d34e19-375c-4f1f-a8ef-5df5966e1776\") " Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933741 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-certs" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.933985 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istiod-ca-cert" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:21.934265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.934217 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-credential-socket" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:21.935026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.934540 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.935026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.934560 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.935026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.934575 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.935026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.934590 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.935026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.934605 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:21.957001 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.944113 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-kube-api-access-vlmsj" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "kube-api-access-vlmsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:21.963169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.963123 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-envoy" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:21.964471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.964417 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-podinfo" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:21.980589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:21.980002 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-token" pod "a0d34e19-375c-4f1f-a8ef-5df5966e1776" (UID: "a0d34e19-375c-4f1f-a8ef-5df5966e1776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:22.024060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.023982 5610 generic.go:356] "Generic (PLEG): container finished" podID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerID="3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9" exitCode=0 Oct 29 22:52:22.024387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.024076 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" Oct 29 22:52:22.024387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.024073 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" event={"ID":"6797f453-aa2d-4d46-8d2d-1a3c4907db03","Type":"ContainerDied","Data":"3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9"} Oct 29 22:52:22.024387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.024229 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9" event={"ID":"6797f453-aa2d-4d46-8d2d-1a3c4907db03","Type":"ContainerDied","Data":"368bf085ba04da6b2643cb8d7539545eef8021be21415d287c96b8ff7ae5cc29"} Oct 29 22:52:22.024387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.024255 5610 scope.go:122] "RemoveContainer" containerID="3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9" Oct 29 22:52:22.027254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.027230 5610 generic.go:356] "Generic (PLEG): container finished" podID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerID="3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31" exitCode=0 Oct 29 22:52:22.027877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.027545 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" event={"ID":"a0d34e19-375c-4f1f-a8ef-5df5966e1776","Type":"ContainerDied","Data":"3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31"} Oct 29 22:52:22.027877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.027596 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" event={"ID":"a0d34e19-375c-4f1f-a8ef-5df5966e1776","Type":"ContainerDied","Data":"55895841d77ef002b73d9718530effa388a3b1f3863e7727b2c0fdbe4e0516eb"} Oct 29 22:52:22.027877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.027670 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7" Oct 29 22:52:22.030202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.029913 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:52:22.030202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.030074 5610 generic.go:356] "Generic (PLEG): container finished" podID="61f84d72-b1ac-49c9-90d6-9a4735e61e4e" containerID="6b3081b52f50bb7a462ff821655a0120c72c2633cfc21434d8c8eca848172fa8" exitCode=0 Oct 29 22:52:22.030202 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.030197 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" event={"ID":"61f84d72-b1ac-49c9-90d6-9a4735e61e4e","Type":"ContainerDied","Data":"6b3081b52f50bb7a462ff821655a0120c72c2633cfc21434d8c8eca848172fa8"} Oct 29 22:52:22.037656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.037631 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-envoy\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.038002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.037980 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istiod-ca-cert\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.038240 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038204 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-socket\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.038401 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038354 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istiod-ca-cert" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:22.038558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038524 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-credential-socket\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.038751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038729 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-socket" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.038867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038730 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-istio-token\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-istio-token\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.038867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038802 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/995527ab-3722-4cbb-858c-137621448ca3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995527ab-3722-4cbb-858c-137621448ca3-istio-podinfo\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.038867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038845 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-kube-api-access-5zs9h\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-kube-api-access-5zs9h\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.039163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038886 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-certs\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.039163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038944 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-socket\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.039163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.038972 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-envoy\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.039163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.039093 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-data\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.039163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.039172 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-data\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.040147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.039221 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-certs\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.040147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.039266 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995527ab-3722-4cbb-858c-137621448ca3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995527ab-3722-4cbb-858c-137621448ca3-istiod-ca-cert\") pod \"995527ab-3722-4cbb-858c-137621448ca3\" (UID: \"995527ab-3722-4cbb-858c-137621448ca3\") " Oct 29 22:52:22.040147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.039299 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-credential-socket\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.040147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.039340 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-podinfo\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.040147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.039414 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-kube-api-access-pgf5s\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-kube-api-access-pgf5s\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.040147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.039469 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-token\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-token\") pod \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\" (UID: \"6797f453-aa2d-4d46-8d2d-1a3c4907db03\") " Oct 29 22:52:22.040510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.040347 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-data" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.040510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.040409 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.040510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.040432 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.040510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.040446 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlmsj\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-kube-api-access-vlmsj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.040510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.040462 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.040510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.040476 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d34e19-375c-4f1f-a8ef-5df5966e1776-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.040840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.040644 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-data" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.040840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.040828 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-certs" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.041265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.041236 5610 generic.go:356] "Generic (PLEG): container finished" podID="995527ab-3722-4cbb-858c-137621448ca3" containerID="fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2" exitCode=0 Oct 29 22:52:22.041357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.041267 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995527ab-3722-4cbb-858c-137621448ca3-istiod-ca-cert" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:22.041357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.041319 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" event={"ID":"995527ab-3722-4cbb-858c-137621448ca3","Type":"ContainerDied","Data":"fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2"} Oct 29 22:52:22.041512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.041360 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" event={"ID":"995527ab-3722-4cbb-858c-137621448ca3","Type":"ContainerDied","Data":"7e42f306f463433d4817cacf942b717b4845a9762aeffc0c16109171ff874e46"} Oct 29 22:52:22.041576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.041541 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-credential-socket" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.041758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.041736 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-credential-socket" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.042025 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.042003 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr" Oct 29 22:52:22.042111 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.042024 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-certs" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.042453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.042431 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-socket" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.044936 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.044707 5610 scope.go:122] "RemoveContainer" containerID="3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9" Oct 29 22:52:22.045426 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:22.045269 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9\": container with ID starting with 3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9 not found: ID does not exist" containerID="3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9" Oct 29 22:52:22.045426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.045304 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9"} err="failed to get container status \"3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9\": rpc error: code = NotFound desc = could not find container \"3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9\": container with ID starting with 3c2a1d6fe4ff252aca4da7f2041f7a2b4db95ce590d5fb9600ab20ee82c120e9 not found: ID does not exist" Oct 29 22:52:22.045426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.045329 5610 scope.go:122] "RemoveContainer" containerID="3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31" Oct 29 22:52:22.047048 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.046718 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-envoy" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.047695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.047669 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-kube-api-access-pgf5s" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "kube-api-access-pgf5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:22.047808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.047712 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-istio-token" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:22.048908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.048874 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-envoy" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:22.049096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.049065 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/995527ab-3722-4cbb-858c-137621448ca3-istio-podinfo" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:22.050640 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.050605 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-kube-api-access-5zs9h" pod "995527ab-3722-4cbb-858c-137621448ca3" (UID: "995527ab-3722-4cbb-858c-137621448ca3"). InnerVolumeSpecName "kube-api-access-5zs9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:22.050829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.050805 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-podinfo" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:22.055656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.055631 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-token" pod "6797f453-aa2d-4d46-8d2d-1a3c4907db03" (UID: "6797f453-aa2d-4d46-8d2d-1a3c4907db03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:22.080083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.080062 5610 scope.go:122] "RemoveContainer" containerID="3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31" Oct 29 22:52:22.080614 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:22.080458 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31\": container with ID starting with 3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31 not found: ID does not exist" containerID="3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31" Oct 29 22:52:22.080614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.080496 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31"} err="failed to get container status \"3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31\": rpc error: code = NotFound desc = could not find container \"3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31\": container with ID starting with 3b7d4ba1650f997523b9cfb4abe25462a24da1701c2a31667678e799e8147a31 not found: ID does not exist" Oct 29 22:52:22.080614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.080521 5610 scope.go:122] "RemoveContainer" containerID="6b3081b52f50bb7a462ff821655a0120c72c2633cfc21434d8c8eca848172fa8" Oct 29 22:52:22.099706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.099545 5610 scope.go:122] "RemoveContainer" containerID="fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2" Oct 29 22:52:22.189473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.177114 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls\") pod \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " Oct 29 22:52:22.207415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.189944 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-metrics-certs\") pod \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " Oct 29 22:52:22.228173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.228134 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-openshift-service-ca\") pod \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.338261 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-kube-api-access-8lwmg\" (UniqueName: \"kubernetes.io/projected/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-kube-api-access-8lwmg\") pod \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\" (UID: \"61f84d72-b1ac-49c9-90d6-9a4735e61e4e\") " Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339127 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339153 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339168 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339183 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339198 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339211 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995527ab-3722-4cbb-858c-137621448ca3-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339226 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zs9h\" (UniqueName: \"kubernetes.io/projected/995527ab-3722-4cbb-858c-137621448ca3-kube-api-access-5zs9h\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339240 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339254 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339267 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339280 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339291 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339303 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995527ab-3722-4cbb-858c-137621448ca3-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339316 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995527ab-3722-4cbb-858c-137621448ca3-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339328 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6797f453-aa2d-4d46-8d2d-1a3c4907db03-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339340 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6797f453-aa2d-4d46-8d2d-1a3c4907db03-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.339354 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgf5s\" (UniqueName: \"kubernetes.io/projected/6797f453-aa2d-4d46-8d2d-1a3c4907db03-kube-api-access-pgf5s\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.288542 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.279666 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-openshift-service-ca" pod "61f84d72-b1ac-49c9-90d6-9a4735e61e4e" (UID: "61f84d72-b1ac-49c9-90d6-9a4735e61e4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.303352 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls" pod "61f84d72-b1ac-49c9-90d6-9a4735e61e4e" (UID: "61f84d72-b1ac-49c9-90d6-9a4735e61e4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:22.399418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.329834 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-metrics-certs" pod "61f84d72-b1ac-49c9-90d6-9a4735e61e4e" (UID: "61f84d72-b1ac-49c9-90d6-9a4735e61e4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:22.436266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.433227 5610 scope.go:122] "RemoveContainer" containerID="fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2" Oct 29 22:52:22.453479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.446929 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7"] Oct 29 22:52:22.453479 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:22.447160 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2\": container with ID starting with fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2 not found: ID does not exist" containerID="fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2" Oct 29 22:52:22.453479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.447253 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2"} err="failed to get container status \"fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2\": rpc error: code = NotFound desc = could not find container \"fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2\": container with ID starting with fe1df6b141db542ebb5c1082e418264b9333bb7a729c884e4f3d3e23f4650ee2 not found: ID does not exist" Oct 29 22:52:22.453983 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.453955 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.454180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.454163 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.454293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.454279 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.454602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.454571 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-kube-api-access-8lwmg" pod "61f84d72-b1ac-49c9-90d6-9a4735e61e4e" (UID: "61f84d72-b1ac-49c9-90d6-9a4735e61e4e"). InnerVolumeSpecName "kube-api-access-8lwmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:22.460292 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.460243 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-5zbv7"] Oct 29 22:52:22.477513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.477486 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:22.559418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.556521 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lwmg\" (UniqueName: \"kubernetes.io/projected/61f84d72-b1ac-49c9-90d6-9a4735e61e4e-kube-api-access-8lwmg\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:22.561959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.561927 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9"] Oct 29 22:52:22.569285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.569210 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x4fl9"] Oct 29 22:52:22.573924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.573894 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr"] Oct 29 22:52:22.578437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.578388 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-pqjzr"] Oct 29 22:52:22.620722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:22.620699 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:23.051330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.051302 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" Oct 29 22:52:23.053408 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.051358 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng" event={"ID":"61f84d72-b1ac-49c9-90d6-9a4735e61e4e","Type":"ContainerDied","Data":"7e6344fb283a9f3d9dcdda8adb22f4a6d3a827bbdd8f4f852dfd85a229e7d5f5"} Oct 29 22:52:23.070587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.070536 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:23.104844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.104816 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng"] Oct 29 22:52:23.110114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.110087 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-379a6d-554f684fd7-2tlng"] Oct 29 22:52:23.638947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.638911 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f84d72-b1ac-49c9-90d6-9a4735e61e4e" path="/var/lib/kubelet/pods/61f84d72-b1ac-49c9-90d6-9a4735e61e4e/volumes" Oct 29 22:52:23.639579 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.639550 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" path="/var/lib/kubelet/pods/6797f453-aa2d-4d46-8d2d-1a3c4907db03/volumes" Oct 29 22:52:23.640026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.640011 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995527ab-3722-4cbb-858c-137621448ca3" path="/var/lib/kubelet/pods/995527ab-3722-4cbb-858c-137621448ca3/volumes" Oct 29 22:52:23.640489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:23.640470 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" path="/var/lib/kubelet/pods/a0d34e19-375c-4f1f-a8ef-5df5966e1776/volumes" Oct 29 22:52:24.227674 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:24.227636 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:25.779429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:25.778293 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:27.689013 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:27.687908 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:29.271481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:29.259716 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:29.727591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:29.727557 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:29.976841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:29.976802 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:30.075766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:30.075674 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b"] Oct 29 22:52:30.076239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:30.076188 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" containerID="cri-o://953b7988109ab1e62ce2c31794a82e1f3c65980616761594fd2ab7bef142d594" gracePeriod=30 Oct 29 22:52:30.260071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:30.259909 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:31.347087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:31.347056 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:31.547280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:31.547240 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8"] Oct 29 22:52:31.682474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:31.661364 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:33.704788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:33.704751 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:34.170476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:34.165622 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:34.335146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:34.335106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:35.251065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.251032 5610 generic.go:356] "Generic (PLEG): container finished" podID="c4578473-cc01-462c-920b-c9e76538ca5e" containerID="953b7988109ab1e62ce2c31794a82e1f3c65980616761594fd2ab7bef142d594" exitCode=0 Oct 29 22:52:35.251573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.251121 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" event={"ID":"c4578473-cc01-462c-920b-c9e76538ca5e","Type":"ContainerDied","Data":"953b7988109ab1e62ce2c31794a82e1f3c65980616761594fd2ab7bef142d594"} Oct 29 22:52:35.251573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.251191 5610 scope.go:122] "RemoveContainer" containerID="96c460cc54a909d9fec5f1973122e2a649c28de502549c41c7e4fc1f1bea015e" Oct 29 22:52:35.399904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.399878 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:52:35.544090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544060 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-kube-api-access-qj7dj\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-kube-api-access-qj7dj\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544151 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-envoy\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544240 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-credential-socket\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544295 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-socket\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544464 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-data\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544530 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-istio-token\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-istio-token\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544535 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-credential-socket" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.544573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544559 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-socket" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.544762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544592 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4578473-cc01-462c-920b-c9e76538ca5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4578473-cc01-462c-920b-c9e76538ca5e-istiod-ca-cert\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544629 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4578473-cc01-462c-920b-c9e76538ca5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4578473-cc01-462c-920b-c9e76538ca5e-istio-podinfo\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544690 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-certs\") pod \"c4578473-cc01-462c-920b-c9e76538ca5e\" (UID: \"c4578473-cc01-462c-920b-c9e76538ca5e\") " Oct 29 22:52:35.544934 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544753 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-data" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.545000 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.544957 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4578473-cc01-462c-920b-c9e76538ca5e-istiod-ca-cert" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:35.545328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.545305 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-certs" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.545973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.545945 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.546095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.545975 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.546095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.545994 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.546095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.546008 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.546095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.546021 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4578473-cc01-462c-920b-c9e76538ca5e-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.546662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.546637 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-kube-api-access-qj7dj" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "kube-api-access-qj7dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:35.546783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.546767 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-envoy" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:35.547004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.546975 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4578473-cc01-462c-920b-c9e76538ca5e-istio-podinfo" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:52:35.547090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.547044 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-istio-token" pod "c4578473-cc01-462c-920b-c9e76538ca5e" (UID: "c4578473-cc01-462c-920b-c9e76538ca5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:35.658429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.658140 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4578473-cc01-462c-920b-c9e76538ca5e-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.658429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.658180 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj7dj\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-kube-api-access-qj7dj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.658429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.658195 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4578473-cc01-462c-920b-c9e76538ca5e-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.658429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.658208 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4578473-cc01-462c-920b-c9e76538ca5e-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:35.683455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:35.678115 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:36.261328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:36.261296 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" Oct 29 22:52:36.261866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:36.261346 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b" event={"ID":"c4578473-cc01-462c-920b-c9e76538ca5e","Type":"ContainerDied","Data":"30619d547ec2ec47390ca33d06b7206b8d27624888bfb73b9d6cd06dc4414575"} Oct 29 22:52:36.261866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:36.261448 5610 scope.go:122] "RemoveContainer" containerID="953b7988109ab1e62ce2c31794a82e1f3c65980616761594fd2ab7bef142d594" Oct 29 22:52:36.320690 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:36.320657 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b"] Oct 29 22:52:36.325976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:36.325947 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-379a6d-openshift-default-7969d87647-zjp5b"] Oct 29 22:52:37.484989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:37.484948 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:37.638974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:37.638942 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" path="/var/lib/kubelet/pods/c4578473-cc01-462c-920b-c9e76538ca5e/volumes" Oct 29 22:52:39.001413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:39.000652 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:39.466590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:39.466527 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:40.285148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:40.285105 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:40.725871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:40.725835 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:41.108979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:41.108890 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:41.257329 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:41.257297 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:42.463265 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:42.463227 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:42.626875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:42.626830 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:42.777256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:42.777178 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:43.713199 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:43.713162 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:52:44.102515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:44.102427 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:45.934416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:45.931037 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:46.570137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.570092 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" podUID="bc290c4e-3d45-4547-bedc-ef18c50947b6" containerName="main" containerID="cri-o://4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c" gracePeriod=45 Oct 29 22:52:46.889149 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.889118 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:52:46.946896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.946859 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-home\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-home\") pod \"bc290c4e-3d45-4547-bedc-ef18c50947b6\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " Oct 29 22:52:46.947442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.947026 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc290c4e-3d45-4547-bedc-ef18c50947b6-kube-api-access-mn52c\" (UniqueName: \"kubernetes.io/projected/bc290c4e-3d45-4547-bedc-ef18c50947b6-kube-api-access-mn52c\") pod \"bc290c4e-3d45-4547-bedc-ef18c50947b6\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " Oct 29 22:52:46.947442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.947096 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-model-cache\") pod \"bc290c4e-3d45-4547-bedc-ef18c50947b6\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " Oct 29 22:52:46.947442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.947168 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-home" pod "bc290c4e-3d45-4547-bedc-ef18c50947b6" (UID: "bc290c4e-3d45-4547-bedc-ef18c50947b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:46.947442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.947240 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc290c4e-3d45-4547-bedc-ef18c50947b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc290c4e-3d45-4547-bedc-ef18c50947b6-tls-certs\") pod \"bc290c4e-3d45-4547-bedc-ef18c50947b6\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " Oct 29 22:52:46.947442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.947380 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-dshm\") pod \"bc290c4e-3d45-4547-bedc-ef18c50947b6\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " Oct 29 22:52:46.947442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.947433 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-tmp-dir\") pod \"bc290c4e-3d45-4547-bedc-ef18c50947b6\" (UID: \"bc290c4e-3d45-4547-bedc-ef18c50947b6\") " Oct 29 22:52:46.947832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.947759 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-model-cache" pod "bc290c4e-3d45-4547-bedc-ef18c50947b6" (UID: "bc290c4e-3d45-4547-bedc-ef18c50947b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:46.948677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.948357 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:46.948677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.948412 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:46.948969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.948944 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-tmp-dir" pod "bc290c4e-3d45-4547-bedc-ef18c50947b6" (UID: "bc290c4e-3d45-4547-bedc-ef18c50947b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:46.950912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.950879 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc290c4e-3d45-4547-bedc-ef18c50947b6-kube-api-access-mn52c" pod "bc290c4e-3d45-4547-bedc-ef18c50947b6" (UID: "bc290c4e-3d45-4547-bedc-ef18c50947b6"). InnerVolumeSpecName "kube-api-access-mn52c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:46.951024 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.950932 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-dshm" pod "bc290c4e-3d45-4547-bedc-ef18c50947b6" (UID: "bc290c4e-3d45-4547-bedc-ef18c50947b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:52:46.952625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:46.952592 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc290c4e-3d45-4547-bedc-ef18c50947b6-tls-certs" pod "bc290c4e-3d45-4547-bedc-ef18c50947b6" (UID: "bc290c4e-3d45-4547-bedc-ef18c50947b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:47.050182 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.050149 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn52c\" (UniqueName: \"kubernetes.io/projected/bc290c4e-3d45-4547-bedc-ef18c50947b6-kube-api-access-mn52c\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:47.050182 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.050179 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc290c4e-3d45-4547-bedc-ef18c50947b6-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:47.050182 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.050188 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:47.050438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.050197 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc290c4e-3d45-4547-bedc-ef18c50947b6-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:47.359214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.359169 5610 generic.go:356] "Generic (PLEG): container finished" podID="bc290c4e-3d45-4547-bedc-ef18c50947b6" containerID="4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c" exitCode=0 Oct 29 22:52:47.359214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.359216 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" Oct 29 22:52:47.359545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.359231 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" event={"ID":"bc290c4e-3d45-4547-bedc-ef18c50947b6","Type":"ContainerDied","Data":"4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c"} Oct 29 22:52:47.359545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.359280 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8" event={"ID":"bc290c4e-3d45-4547-bedc-ef18c50947b6","Type":"ContainerDied","Data":"8b5e96a56634eef5420c630247733b2822455f695ccf85e87055f9ab15c57ccb"} Oct 29 22:52:47.359545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.359304 5610 scope.go:122] "RemoveContainer" containerID="4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c" Oct 29 22:52:47.372428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.372374 5610 scope.go:122] "RemoveContainer" containerID="4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c" Oct 29 22:52:47.372730 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:47.372700 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c\": container with ID starting with 4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c not found: ID does not exist" containerID="4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c" Oct 29 22:52:47.372832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.372738 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c"} err="failed to get container status \"4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c\": rpc error: code = NotFound desc = could not find container \"4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c\": container with ID starting with 4e20704f8f14557cee152ca9e43503685b1b650638cedf157af0d93f708cad5c not found: ID does not exist" Oct 29 22:52:47.414899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.414874 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8"] Oct 29 22:52:47.420073 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.420050 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8ebc60/test-model-8ebc60-kserve-6f745f57ff-dvvp8"] Oct 29 22:52:47.515007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.514958 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:47.640774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:47.640698 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc290c4e-3d45-4547-bedc-ef18c50947b6" path="/var/lib/kubelet/pods/bc290c4e-3d45-4547-bedc-ef18c50947b6/volumes" Oct 29 22:52:48.729252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:48.729219 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq"] Oct 29 22:52:48.729790 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:48.729481 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" podUID="1fcd01ef-3c83-405a-9333-f2bd4d9cf361" containerName="maas-api" containerID="cri-o://41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef" gracePeriod=30 Oct 29 22:52:48.835728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:48.835688 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:49.117171 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.117079 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt"] Oct 29 22:52:49.117648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.117615 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" podUID="1f326efd-a274-4b02-98f5-b2ca39ca6209" containerName="maas-api" containerID="cri-o://a26ba01a5b67948abdf36786ba02be43e10d275401abc91d97d6d0aa589d28e5" gracePeriod=30 Oct 29 22:52:49.284661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.225527 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:49.565418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.562561 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:49.673505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.670973 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:52:49.743383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.721072 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:49.743383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.721321 5610 generic.go:356] "Generic (PLEG): container finished" podID="1f326efd-a274-4b02-98f5-b2ca39ca6209" containerID="a26ba01a5b67948abdf36786ba02be43e10d275401abc91d97d6d0aa589d28e5" exitCode=0 Oct 29 22:52:49.743383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.724558 5610 generic.go:356] "Generic (PLEG): container finished" podID="1fcd01ef-3c83-405a-9333-f2bd4d9cf361" containerID="41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef" exitCode=0 Oct 29 22:52:49.743383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.724648 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" Oct 29 22:52:49.743383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.728596 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" event={"ID":"1f326efd-a274-4b02-98f5-b2ca39ca6209","Type":"ContainerDied","Data":"a26ba01a5b67948abdf36786ba02be43e10d275401abc91d97d6d0aa589d28e5"} Oct 29 22:52:49.743383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.728651 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" event={"ID":"1fcd01ef-3c83-405a-9333-f2bd4d9cf361","Type":"ContainerDied","Data":"41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef"} Oct 29 22:52:49.743383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.728683 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq" event={"ID":"1fcd01ef-3c83-405a-9333-f2bd4d9cf361","Type":"ContainerDied","Data":"a26d420152c33b277bdaf9f9d70da40c50161df9b4cf95ac9065b8f5e4762b2f"} Oct 29 22:52:49.743383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.728710 5610 scope.go:122] "RemoveContainer" containerID="41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef" Oct 29 22:52:49.803424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.802970 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-kube-api-access-v692l\" (UniqueName: \"kubernetes.io/projected/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-kube-api-access-v692l\") pod \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " Oct 29 22:52:49.803424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.803260 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-openshift-service-ca\") pod \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " Oct 29 22:52:49.803424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.803331 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-metrics-certs\") pod \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " Oct 29 22:52:49.803424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.803411 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-maas-api-tls\") pod \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\" (UID: \"1fcd01ef-3c83-405a-9333-f2bd4d9cf361\") " Oct 29 22:52:49.815468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.815437 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-openshift-service-ca" pod "1fcd01ef-3c83-405a-9333-f2bd4d9cf361" (UID: "1fcd01ef-3c83-405a-9333-f2bd4d9cf361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:49.827970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.827930 5610 scope.go:122] "RemoveContainer" containerID="41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef" Oct 29 22:52:49.830282 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:52:49.830258 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef\": container with ID starting with 41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef not found: ID does not exist" containerID="41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef" Oct 29 22:52:49.830537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.830509 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef"} err="failed to get container status \"41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef\": rpc error: code = NotFound desc = could not find container \"41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef\": container with ID starting with 41b33243b580fdee0a8d8f441c6ad36aa4d2b376b7df021adca0088b7e4e54ef not found: ID does not exist" Oct 29 22:52:49.838575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.838547 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-maas-api-tls" pod "1fcd01ef-3c83-405a-9333-f2bd4d9cf361" (UID: "1fcd01ef-3c83-405a-9333-f2bd4d9cf361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:49.845651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.841582 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-kube-api-access-v692l" pod "1fcd01ef-3c83-405a-9333-f2bd4d9cf361" (UID: "1fcd01ef-3c83-405a-9333-f2bd4d9cf361"). InnerVolumeSpecName "kube-api-access-v692l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:49.846904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.846877 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-metrics-certs" pod "1fcd01ef-3c83-405a-9333-f2bd4d9cf361" (UID: "1fcd01ef-3c83-405a-9333-f2bd4d9cf361"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:49.975343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.975300 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:49.975554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.975349 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:49.975554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.975366 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:49.975554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:49.975402 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v692l\" (UniqueName: \"kubernetes.io/projected/1fcd01ef-3c83-405a-9333-f2bd4d9cf361-kube-api-access-v692l\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:50.138916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.138880 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq"] Oct 29 22:52:50.142635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.142595 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6bc7b2-54ffd6dc5-fxbbq"] Oct 29 22:52:50.155228 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.155193 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:50.238154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.238091 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:52:50.306293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.306260 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:50.338299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.338262 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f326efd-a274-4b02-98f5-b2ca39ca6209-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f326efd-a274-4b02-98f5-b2ca39ca6209-openshift-service-ca\") pod \"1f326efd-a274-4b02-98f5-b2ca39ca6209\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " Oct 29 22:52:50.338540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.338359 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-metrics-certs\") pod \"1f326efd-a274-4b02-98f5-b2ca39ca6209\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " Oct 29 22:52:50.338540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.338456 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-maas-api-tls\") pod \"1f326efd-a274-4b02-98f5-b2ca39ca6209\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " Oct 29 22:52:50.338698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.338622 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f326efd-a274-4b02-98f5-b2ca39ca6209-openshift-service-ca" pod "1f326efd-a274-4b02-98f5-b2ca39ca6209" (UID: "1f326efd-a274-4b02-98f5-b2ca39ca6209"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:52:50.338761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.338719 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f326efd-a274-4b02-98f5-b2ca39ca6209-kube-api-access-xdgng\" (UniqueName: \"kubernetes.io/projected/1f326efd-a274-4b02-98f5-b2ca39ca6209-kube-api-access-xdgng\") pod \"1f326efd-a274-4b02-98f5-b2ca39ca6209\" (UID: \"1f326efd-a274-4b02-98f5-b2ca39ca6209\") " Oct 29 22:52:50.339291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.339270 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f326efd-a274-4b02-98f5-b2ca39ca6209-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:50.340623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.340601 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-maas-api-tls" pod "1f326efd-a274-4b02-98f5-b2ca39ca6209" (UID: "1f326efd-a274-4b02-98f5-b2ca39ca6209"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:50.340736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.340639 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-metrics-certs" pod "1f326efd-a274-4b02-98f5-b2ca39ca6209" (UID: "1f326efd-a274-4b02-98f5-b2ca39ca6209"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:52:50.340736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.340660 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f326efd-a274-4b02-98f5-b2ca39ca6209-kube-api-access-xdgng" pod "1f326efd-a274-4b02-98f5-b2ca39ca6209" (UID: "1f326efd-a274-4b02-98f5-b2ca39ca6209"). InnerVolumeSpecName "kube-api-access-xdgng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:52:50.440893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.440856 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:50.440893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.440887 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdgng\" (UniqueName: \"kubernetes.io/projected/1f326efd-a274-4b02-98f5-b2ca39ca6209-kube-api-access-xdgng\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:50.440893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.440897 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f326efd-a274-4b02-98f5-b2ca39ca6209-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:52:50.478325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.477383 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:50.733545 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.733513 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" Oct 29 22:52:50.733728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.733568 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt" event={"ID":"1f326efd-a274-4b02-98f5-b2ca39ca6209","Type":"ContainerDied","Data":"7de1345a35e25f4ef8c8085e6e7dd1b7c60e2fa7fc7291bddf0b96876c3a5727"} Oct 29 22:52:50.733728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.733631 5610 scope.go:122] "RemoveContainer" containerID="a26ba01a5b67948abdf36786ba02be43e10d275401abc91d97d6d0aa589d28e5" Oct 29 22:52:50.741376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.741342 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:50.782345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.782315 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt"] Oct 29 22:52:50.785987 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.785962 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-172c10-576c54476c-twdwt"] Oct 29 22:52:50.915219 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:50.915186 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:51.637519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:51.637483 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f326efd-a274-4b02-98f5-b2ca39ca6209" path="/var/lib/kubelet/pods/1f326efd-a274-4b02-98f5-b2ca39ca6209/volumes" Oct 29 22:52:51.638119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:51.638095 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fcd01ef-3c83-405a-9333-f2bd4d9cf361" path="/var/lib/kubelet/pods/1fcd01ef-3c83-405a-9333-f2bd4d9cf361/volumes" Oct 29 22:52:52.168354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:52.158018 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:53.903421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:53.897329 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:55.454414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:55.450504 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:56.873341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:56.873305 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:57.039375 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.039342 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:57.188525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.150892 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs"] Oct 29 22:52:57.188525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.153344 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" containerID="cri-o://060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649" gracePeriod=30 Oct 29 22:52:57.191547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.191509 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p"] Oct 29 22:52:57.191961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.191926 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" containerID="cri-o://d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72" gracePeriod=30 Oct 29 22:52:57.203881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.202609 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4"] Oct 29 22:52:57.203881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.202859 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" containerID="cri-o://2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67" gracePeriod=30 Oct 29 22:52:57.220421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.213003 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:57.220421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.220365 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm"] Oct 29 22:52:57.220647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.220602 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" containerID="cri-o://93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7" gracePeriod=30 Oct 29 22:52:57.406413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:57.406081 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:58.008841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:58.008807 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:58.212603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:58.212566 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:58.212787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:58.212628 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:52:58.236457 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:58.236423 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:58.425660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:58.425570 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8"] Oct 29 22:52:58.425841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:58.425813 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" containerID="cri-o://0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7" gracePeriod=30 Oct 29 22:52:58.470964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:58.470929 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:59.132031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:59.131994 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:52:59.208759 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:59.208721 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:52:59.208950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:52:59.208799 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:53:00.111818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:00.111784 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:53:00.111994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:00.111864 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:53:00.227855 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:00.227818 5610 patch_prober.go:29] interesting pod/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:53:00.228310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:00.227889 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:53:01.077217 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:01.077178 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:02.423937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.417301 5610 patch_prober.go:29] interesting pod/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:53:02.423937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.417360 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:53:02.443295 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.443135 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:53:02.515259 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515227 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-token\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-token\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.515432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515322 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-certs\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.515494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515461 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-credential-socket\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.515589 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515558 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-podinfo\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.515655 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515628 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istiod-ca-cert\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.515715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515671 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-certs" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.515715 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515680 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-socket\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.515787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515724 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-kube-api-access-w8jnv\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-kube-api-access-w8jnv\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.515787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515766 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-envoy\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.515877 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515807 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-credential-socket" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.515916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.515882 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-data\") pod \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\" (UID: \"b3dbba23-277e-46f6-9050-7ceb0682b2b9\") " Oct 29 22:53:02.516681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.516653 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.516818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.516683 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.516926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.516906 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-data" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.517085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.517066 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-socket" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.517330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.517309 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istiod-ca-cert" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.518351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.518298 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-envoy" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.518579 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.518552 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-podinfo" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.519350 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.519321 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-kube-api-access-w8jnv" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "kube-api-access-w8jnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.519492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.519463 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-token" pod "b3dbba23-277e-46f6-9050-7ceb0682b2b9" (UID: "b3dbba23-277e-46f6-9050-7ceb0682b2b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.530230 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.530201 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:53:02.614443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.614420 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:53:02.618083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618058 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-token\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-token\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618112 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-kube-api-access-gpdz2\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-kube-api-access-gpdz2\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618155 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-data\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618292 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-envoy\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618325 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-credential-socket\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618383 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-podinfo\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618485 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-certs\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618528 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istiod-ca-cert\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618525 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-data" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.618776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618684 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-socket\") pod \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\" (UID: \"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10\") " Oct 29 22:53:02.618776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.618751 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-credential-socket" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.619125 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619094 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-certs" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.619306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619283 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-socket" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619414 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619431 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619445 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619456 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619464 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619475 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8jnv\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-kube-api-access-w8jnv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619483 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619491 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619499 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619510 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dbba23-277e-46f6-9050-7ceb0682b2b9-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619522 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.619850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.619743 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istiod-ca-cert" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.621260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.621234 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-kube-api-access-gpdz2" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "kube-api-access-gpdz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.622184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.622144 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-podinfo" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.622484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.622460 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-envoy" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.622979 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.622954 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-token" pod "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" (UID: "788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.624678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.624662 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:53:02.720355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.720272 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-envoy\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.720991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.720963 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-kube-api-access-zkxfm\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-kube-api-access-zkxfm\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721255 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa28810-d8d4-4027-b974-fd3b433a46a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa28810-d8d4-4027-b974-fd3b433a46a2-istiod-ca-cert\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721361 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-istio-token\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-istio-token\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721438 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-podinfo\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721489 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-credential-socket\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721699 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa28810-d8d4-4027-b974-fd3b433a46a2-istiod-ca-cert" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721755 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be592e7c-e303-428e-8f0b-906554d7b96b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be592e7c-e303-428e-8f0b-906554d7b96b-istiod-ca-cert\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721813 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-certs\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721914 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-envoy\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.721977 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-kube-api-access-jgkq7\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-kube-api-access-jgkq7\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722033 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-credential-socket\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.722347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722078 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-certs\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722719 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-socket\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722752 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-token\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-token\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722795 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-data\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722820 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be592e7c-e303-428e-8f0b-906554d7b96b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be592e7c-e303-428e-8f0b-906554d7b96b-istio-podinfo\") pod \"be592e7c-e303-428e-8f0b-906554d7b96b\" (UID: \"be592e7c-e303-428e-8f0b-906554d7b96b\") " Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722856 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-socket\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722902 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-data\") pod \"7aa28810-d8d4-4027-b974-fd3b433a46a2\" (UID: \"7aa28810-d8d4-4027-b974-fd3b433a46a2\") " Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.722949 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-credential-socket" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.723200 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-socket" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.723768 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be592e7c-e303-428e-8f0b-906554d7b96b-istiod-ca-cert" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.723990 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-credential-socket" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.724198 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-certs" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.724351 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-certs" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.726051 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.726284 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.726504 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdz2\" (UniqueName: \"kubernetes.io/projected/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-kube-api-access-gpdz2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.727087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.726673 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be592e7c-e303-428e-8f0b-906554d7b96b-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.728071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.727108 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.729330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.728998 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.729330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.729023 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.729330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.729039 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.729330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.729054 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.729330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.729067 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.729330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.729079 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.729330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.729092 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa28810-d8d4-4027-b974-fd3b433a46a2-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.731146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.730410 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-socket" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.731146 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.730604 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-data" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.733411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.733367 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-data" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.738813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.738791 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-istio-token" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.738923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.738911 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-kube-api-access-zkxfm" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "kube-api-access-zkxfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.739040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.739024 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-envoy" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.739131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.739115 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-envoy" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:02.739201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.739184 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be592e7c-e303-428e-8f0b-906554d7b96b-istio-podinfo" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.739548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.739525 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-kube-api-access-jgkq7" pod "be592e7c-e303-428e-8f0b-906554d7b96b" (UID: "be592e7c-e303-428e-8f0b-906554d7b96b"). InnerVolumeSpecName "kube-api-access-jgkq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.740374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.740354 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-podinfo" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:02.740662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.740637 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-token" pod "7aa28810-d8d4-4027-b974-fd3b433a46a2" (UID: "7aa28810-d8d4-4027-b974-fd3b433a46a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:02.801711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.801503 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.835924 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.835963 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxfm\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-kube-api-access-zkxfm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.835979 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.835995 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.836008 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.836022 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgkq7\" (UniqueName: \"kubernetes.io/projected/be592e7c-e303-428e-8f0b-906554d7b96b-kube-api-access-jgkq7\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.836035 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.836048 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be592e7c-e303-428e-8f0b-906554d7b96b-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.836060 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be592e7c-e303-428e-8f0b-906554d7b96b-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.836072 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.855951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.836086 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa28810-d8d4-4027-b974-fd3b433a46a2-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:02.864886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.863232 5610 generic.go:356] "Generic (PLEG): container finished" podID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerID="2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67" exitCode=0 Oct 29 22:53:02.864886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.863415 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" event={"ID":"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10","Type":"ContainerDied","Data":"2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67"} Oct 29 22:53:02.864886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.863473 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" event={"ID":"788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10","Type":"ContainerDied","Data":"d79a4eef3f3a5181d5c87cc44667928ad1f24d1f892f92adc08ab717a5f11b3f"} Oct 29 22:53:02.864886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.863503 5610 scope.go:122] "RemoveContainer" containerID="2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67" Oct 29 22:53:02.864886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.863659 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4" Oct 29 22:53:02.883519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.874743 5610 generic.go:356] "Generic (PLEG): container finished" podID="be592e7c-e303-428e-8f0b-906554d7b96b" containerID="d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72" exitCode=0 Oct 29 22:53:02.883519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.874871 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" Oct 29 22:53:02.883519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.875689 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" event={"ID":"be592e7c-e303-428e-8f0b-906554d7b96b","Type":"ContainerDied","Data":"d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72"} Oct 29 22:53:02.883519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.875741 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p" event={"ID":"be592e7c-e303-428e-8f0b-906554d7b96b","Type":"ContainerDied","Data":"28dfbadf137b6465aaad9b65338eb9afc96956d1e510f8b4cbfbd23b689e06ef"} Oct 29 22:53:02.899500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.898203 5610 scope.go:122] "RemoveContainer" containerID="2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67" Oct 29 22:53:02.899500 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:02.898963 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67\": container with ID starting with 2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67 not found: ID does not exist" containerID="2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67" Oct 29 22:53:02.899500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.899007 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67"} err="failed to get container status \"2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67\": rpc error: code = NotFound desc = could not find container \"2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67\": container with ID starting with 2322a8fd91963156c4fa79764f5e507f791be9da9d5b6de96cecf4ba69559c67 not found: ID does not exist" Oct 29 22:53:02.899500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.899031 5610 scope.go:122] "RemoveContainer" containerID="d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72" Oct 29 22:53:02.899795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.899608 5610 generic.go:356] "Generic (PLEG): container finished" podID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerID="060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649" exitCode=0 Oct 29 22:53:02.899795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.899690 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" event={"ID":"b3dbba23-277e-46f6-9050-7ceb0682b2b9","Type":"ContainerDied","Data":"060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649"} Oct 29 22:53:02.899795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.899735 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" event={"ID":"b3dbba23-277e-46f6-9050-7ceb0682b2b9","Type":"ContainerDied","Data":"9fbc205f2f1a37a062cd2d62d5ffdf75c5b02a9f82c3749267466cfed53c910e"} Oct 29 22:53:02.906489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.899980 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs" Oct 29 22:53:02.907991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.907963 5610 generic.go:356] "Generic (PLEG): container finished" podID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerID="93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7" exitCode=0 Oct 29 22:53:02.908163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.908146 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" event={"ID":"7aa28810-d8d4-4027-b974-fd3b433a46a2","Type":"ContainerDied","Data":"93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7"} Oct 29 22:53:02.908226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.908190 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" event={"ID":"7aa28810-d8d4-4027-b974-fd3b433a46a2","Type":"ContainerDied","Data":"2fce582104a41261e0723e7380fef871e82a9c6b838ee36ab60828ddc328077b"} Oct 29 22:53:02.908426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.908409 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm" Oct 29 22:53:02.921254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.921234 5610 scope.go:122] "RemoveContainer" containerID="d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72" Oct 29 22:53:02.923685 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:02.921749 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72\": container with ID starting with d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72 not found: ID does not exist" containerID="d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72" Oct 29 22:53:02.923685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.921788 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72"} err="failed to get container status \"d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72\": rpc error: code = NotFound desc = could not find container \"d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72\": container with ID starting with d1f2804939ffb47e58b4d2c2df5943ae3cc2e7edb89685a36ff53919d5177e72 not found: ID does not exist" Oct 29 22:53:02.923685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.921808 5610 scope.go:122] "RemoveContainer" containerID="060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649" Oct 29 22:53:02.933750 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.933727 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4"] Oct 29 22:53:02.936422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.936405 5610 scope.go:122] "RemoveContainer" containerID="060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649" Oct 29 22:53:02.936726 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:02.936695 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649\": container with ID starting with 060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649 not found: ID does not exist" containerID="060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649" Oct 29 22:53:02.936806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.936743 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649"} err="failed to get container status \"060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649\": rpc error: code = NotFound desc = could not find container \"060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649\": container with ID starting with 060e751546253cc8af18fed4a55890ca482f3a2cf4de17378ec8cd2556ed2649 not found: ID does not exist" Oct 29 22:53:02.936806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.936765 5610 scope.go:122] "RemoveContainer" containerID="93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7" Oct 29 22:53:02.939727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.939707 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-tjnt4"] Oct 29 22:53:02.948908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.948893 5610 scope.go:122] "RemoveContainer" containerID="93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7" Oct 29 22:53:02.949176 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:02.949155 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7\": container with ID starting with 93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7 not found: ID does not exist" containerID="93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7" Oct 29 22:53:02.949227 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.949182 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7"} err="failed to get container status \"93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7\": rpc error: code = NotFound desc = could not find container \"93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7\": container with ID starting with 93c8363ce66be4d873d56bb4f21ba8f2fc36717aec8a1fb1fce18dcaeee140f7 not found: ID does not exist" Oct 29 22:53:02.969851 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:02.969819 5610 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe592e7c_e303_428e_8f0b_906554d7b96b.slice/crio-28dfbadf137b6465aaad9b65338eb9afc96956d1e510f8b4cbfbd23b689e06ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788d8d73_dc49_4bc5_aac2_1a8cbf4e5d10.slice/crio-d79a4eef3f3a5181d5c87cc44667928ad1f24d1f892f92adc08ab717a5f11b3f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788d8d73_dc49_4bc5_aac2_1a8cbf4e5d10.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dbba23_277e_46f6_9050_7ceb0682b2b9.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:53:02.971360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.971300 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm"] Oct 29 22:53:02.985208 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:02.985185 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-2lngm"] Oct 29 22:53:03.017663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.017639 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p"] Oct 29 22:53:03.020628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.020604 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-g5x9p"] Oct 29 22:53:03.050600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.050578 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs"] Oct 29 22:53:03.054424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.054370 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6bc7b2-openshift-default-57d46999c5-qqwzs"] Oct 29 22:53:03.641719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.641684 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" path="/var/lib/kubelet/pods/788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10/volumes" Oct 29 22:53:03.642438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.642419 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" path="/var/lib/kubelet/pods/7aa28810-d8d4-4027-b974-fd3b433a46a2/volumes" Oct 29 22:53:03.643066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.643047 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" path="/var/lib/kubelet/pods/b3dbba23-277e-46f6-9050-7ceb0682b2b9/volumes" Oct 29 22:53:03.644128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.644109 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" path="/var/lib/kubelet/pods/be592e7c-e303-428e-8f0b-906554d7b96b/volumes" Oct 29 22:53:03.725695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.725671 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:53:03.849695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849665 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-credential-socket\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.849884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849704 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-envoy\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.849884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849731 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-socket\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.849884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849800 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-istio-token\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-istio-token\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.849884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849823 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-kube-api-access-fwwxc\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-kube-api-access-fwwxc\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.849884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849851 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-data\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.849884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849870 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f3562d-4be8-436f-8435-f972c3e98761-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f3562d-4be8-436f-8435-f972c3e98761-istiod-ca-cert\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.850226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849926 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-credential-socket" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.850226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.849972 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-certs\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.850226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.850061 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-socket" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.850226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.850072 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90f3562d-4be8-436f-8435-f972c3e98761-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f3562d-4be8-436f-8435-f972c3e98761-istio-podinfo\") pod \"90f3562d-4be8-436f-8435-f972c3e98761\" (UID: \"90f3562d-4be8-436f-8435-f972c3e98761\") " Oct 29 22:53:03.850440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.850254 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-data" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.850440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.850372 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f3562d-4be8-436f-8435-f972c3e98761-istiod-ca-cert" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:03.850848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.850826 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-certs" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.850985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.850962 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.851043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.851001 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.851043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.851017 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.851043 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.851031 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f3562d-4be8-436f-8435-f972c3e98761-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.851162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.851044 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.852433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.852367 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-istio-token" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:03.852562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.852542 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-envoy" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:03.852562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.852553 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-kube-api-access-fwwxc" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "kube-api-access-fwwxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:03.852684 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.852623 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90f3562d-4be8-436f-8435-f972c3e98761-istio-podinfo" pod "90f3562d-4be8-436f-8435-f972c3e98761" (UID: "90f3562d-4be8-436f-8435-f972c3e98761"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:03.920993 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.920957 5610 generic.go:356] "Generic (PLEG): container finished" podID="90f3562d-4be8-436f-8435-f972c3e98761" containerID="0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7" exitCode=0 Oct 29 22:53:03.921165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.921011 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" Oct 29 22:53:03.921165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.921031 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" event={"ID":"90f3562d-4be8-436f-8435-f972c3e98761","Type":"ContainerDied","Data":"0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7"} Oct 29 22:53:03.921165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.921082 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8" event={"ID":"90f3562d-4be8-436f-8435-f972c3e98761","Type":"ContainerDied","Data":"3835510dc58938848466783fde842f3a39b4d875b39c542d19a12578a48c4a23"} Oct 29 22:53:03.921165 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.921106 5610 scope.go:122] "RemoveContainer" containerID="0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7" Oct 29 22:53:03.932867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.932849 5610 scope.go:122] "RemoveContainer" containerID="0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7" Oct 29 22:53:03.933179 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:03.933159 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7\": container with ID starting with 0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7 not found: ID does not exist" containerID="0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7" Oct 29 22:53:03.933237 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.933189 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7"} err="failed to get container status \"0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7\": rpc error: code = NotFound desc = could not find container \"0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7\": container with ID starting with 0c6216b8288a21ad8ee8706574895023f35e367b20f97b01422a33b3ec80aee7 not found: ID does not exist" Oct 29 22:53:03.951990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.951961 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f3562d-4be8-436f-8435-f972c3e98761-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.951990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.951986 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.952134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.952002 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwwxc\" (UniqueName: \"kubernetes.io/projected/90f3562d-4be8-436f-8435-f972c3e98761-kube-api-access-fwwxc\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.952134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.952017 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f3562d-4be8-436f-8435-f972c3e98761-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:03.968382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.968339 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8"] Oct 29 22:53:03.971798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:03.971776 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-172c10-openshift-default-6766bf6875-rr2l8"] Oct 29 22:53:04.349468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:04.346778 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:05.637421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:05.637365 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f3562d-4be8-436f-8435-f972c3e98761" path="/var/lib/kubelet/pods/90f3562d-4be8-436f-8435-f972c3e98761/volumes" Oct 29 22:53:05.978233 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:05.978185 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:07.592417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:07.585986 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:08.467530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:08.467464 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:08.610062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:08.610029 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:09.075433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:09.075210 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc"] Oct 29 22:53:09.075850 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:09.075815 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" podUID="2bdc9534-851a-4f75-8094-9d892edea71a" containerName="maas-api" containerID="cri-o://2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2" gracePeriod=30 Oct 29 22:53:09.221635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:09.221599 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:09.985517 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:09.985477 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:53:10.045200 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.045168 5610 generic.go:356] "Generic (PLEG): container finished" podID="2bdc9534-851a-4f75-8094-9d892edea71a" containerID="2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2" exitCode=0 Oct 29 22:53:10.045511 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.045490 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" Oct 29 22:53:10.045669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.045650 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" event={"ID":"2bdc9534-851a-4f75-8094-9d892edea71a","Type":"ContainerDied","Data":"2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2"} Oct 29 22:53:10.045801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.045785 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc" event={"ID":"2bdc9534-851a-4f75-8094-9d892edea71a","Type":"ContainerDied","Data":"9ef27d952419b0ad1a2285b3dee27a78b176a2ebae1bfdbfbf0f01b64f37a1c2"} Oct 29 22:53:10.045941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.045927 5610 scope.go:122] "RemoveContainer" containerID="2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2" Oct 29 22:53:10.053619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.053593 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:10.081460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.081437 5610 scope.go:122] "RemoveContainer" containerID="2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2" Oct 29 22:53:10.082437 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:10.081999 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2\": container with ID starting with 2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2 not found: ID does not exist" containerID="2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2" Oct 29 22:53:10.082437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.082039 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2"} err="failed to get container status \"2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2\": rpc error: code = NotFound desc = could not find container \"2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2\": container with ID starting with 2571b5e8a183eefe80eb4bff5fb872941ffda9b2f2de071abd9176017dfc35b2 not found: ID does not exist" Oct 29 22:53:10.135413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.125372 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-metrics-certs\") pod \"2bdc9534-851a-4f75-8094-9d892edea71a\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " Oct 29 22:53:10.135413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.126035 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bdc9534-851a-4f75-8094-9d892edea71a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc9534-851a-4f75-8094-9d892edea71a-openshift-service-ca\") pod \"2bdc9534-851a-4f75-8094-9d892edea71a\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " Oct 29 22:53:10.135413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.126172 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdc9534-851a-4f75-8094-9d892edea71a-kube-api-access-4mx4b\" (UniqueName: \"kubernetes.io/projected/2bdc9534-851a-4f75-8094-9d892edea71a-kube-api-access-4mx4b\") pod \"2bdc9534-851a-4f75-8094-9d892edea71a\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " Oct 29 22:53:10.135413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.126213 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-maas-api-tls\") pod \"2bdc9534-851a-4f75-8094-9d892edea71a\" (UID: \"2bdc9534-851a-4f75-8094-9d892edea71a\") " Oct 29 22:53:10.135413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.130577 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdc9534-851a-4f75-8094-9d892edea71a-openshift-service-ca" pod "2bdc9534-851a-4f75-8094-9d892edea71a" (UID: "2bdc9534-851a-4f75-8094-9d892edea71a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:10.135822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.135673 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdc9534-851a-4f75-8094-9d892edea71a-kube-api-access-4mx4b" pod "2bdc9534-851a-4f75-8094-9d892edea71a" (UID: "2bdc9534-851a-4f75-8094-9d892edea71a"). InnerVolumeSpecName "kube-api-access-4mx4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:10.135888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.135822 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-metrics-certs" pod "2bdc9534-851a-4f75-8094-9d892edea71a" (UID: "2bdc9534-851a-4f75-8094-9d892edea71a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:10.139450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.137716 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-maas-api-tls" pod "2bdc9534-851a-4f75-8094-9d892edea71a" (UID: "2bdc9534-851a-4f75-8094-9d892edea71a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:10.226857 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.226763 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:10.227695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.227667 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:10.227695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.227692 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdc9534-851a-4f75-8094-9d892edea71a-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:10.227916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.227706 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mx4b\" (UniqueName: \"kubernetes.io/projected/2bdc9534-851a-4f75-8094-9d892edea71a-kube-api-access-4mx4b\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:10.227916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.227722 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bdc9534-851a-4f75-8094-9d892edea71a-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:10.389974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.389939 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc"] Oct 29 22:53:10.393697 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.393663 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:10.400502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.400480 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ad69b4-7dcc4bb9d8-tmnnc"] Oct 29 22:53:10.579808 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.579733 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:10.743269 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:10.743237 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:11.639796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:11.639763 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdc9534-851a-4f75-8094-9d892edea71a" path="/var/lib/kubelet/pods/2bdc9534-851a-4f75-8094-9d892edea71a/volumes" Oct 29 22:53:12.025469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:12.025431 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:13.735941 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:13.735770 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:13.765337 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:13.765298 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:53:15.251415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:15.247961 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:17.008503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:17.008456 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:17.173794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:17.173759 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:17.414459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:17.414329 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr"] Oct 29 22:53:17.414635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:17.414605 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" containerID="cri-o://c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b" gracePeriod=30 Oct 29 22:53:17.542428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:17.540028 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:18.452314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.452268 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:18.648131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.648098 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:18.823912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.823834 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg"] Oct 29 22:53:18.824828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824807 5610 cpu_manager.go:414] "containerMap: removing container" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" Oct 29 22:53:18.824828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824826 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824854 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1fcd01ef-3c83-405a-9333-f2bd4d9cf361" containerName="maas-api" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824859 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fcd01ef-3c83-405a-9333-f2bd4d9cf361" containerName="maas-api" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824867 5610 cpu_manager.go:414] "containerMap: removing container" podUID="089f646b-d23c-481a-b1e3-19fce519cd91" containerName="main" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824872 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089f646b-d23c-481a-b1e3-19fce519cd91" containerName="main" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824880 5610 cpu_manager.go:414] "containerMap: removing container" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824885 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824894 5610 cpu_manager.go:414] "containerMap: removing container" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824899 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824907 5610 cpu_manager.go:414] "containerMap: removing container" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824911 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824922 5610 cpu_manager.go:414] "containerMap: removing container" podUID="bc290c4e-3d45-4547-bedc-ef18c50947b6" containerName="main" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824926 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc290c4e-3d45-4547-bedc-ef18c50947b6" containerName="main" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824940 5610 cpu_manager.go:414] "containerMap: removing container" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824944 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824957 5610 cpu_manager.go:414] "containerMap: removing container" podUID="7ad9976c-7a0c-468b-b3f3-16fc3a567eda" containerName="main" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824964 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad9976c-7a0c-468b-b3f3-16fc3a567eda" containerName="main" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824976 5610 cpu_manager.go:414] "containerMap: removing container" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" Oct 29 22:53:18.824977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824981 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824991 5610 cpu_manager.go:414] "containerMap: removing container" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.824996 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825004 5610 cpu_manager.go:414] "containerMap: removing container" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825008 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825017 5610 cpu_manager.go:414] "containerMap: removing container" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825022 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825031 5610 cpu_manager.go:414] "containerMap: removing container" podUID="2bdc9534-851a-4f75-8094-9d892edea71a" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825035 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdc9534-851a-4f75-8094-9d892edea71a" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825043 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ecb5ed71-07f9-4334-aab8-cef607b23d5d" containerName="main" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825048 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb5ed71-07f9-4334-aab8-cef607b23d5d" containerName="main" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825055 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825059 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825065 5610 cpu_manager.go:414] "containerMap: removing container" podUID="61f84d72-b1ac-49c9-90d6-9a4735e61e4e" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825069 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f84d72-b1ac-49c9-90d6-9a4735e61e4e" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825075 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1f326efd-a274-4b02-98f5-b2ca39ca6209" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825079 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f326efd-a274-4b02-98f5-b2ca39ca6209" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825204 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="6797f453-aa2d-4d46-8d2d-1a3c4907db03" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825211 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad9976c-7a0c-468b-b3f3-16fc3a567eda" containerName="main" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825218 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825224 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d34e19-375c-4f1f-a8ef-5df5966e1776" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825231 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="995527ab-3722-4cbb-858c-137621448ca3" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825239 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdc9534-851a-4f75-8094-9d892edea71a" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825244 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb5ed71-07f9-4334-aab8-cef607b23d5d" containerName="main" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825250 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dbba23-277e-46f6-9050-7ceb0682b2b9" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825255 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f84d72-b1ac-49c9-90d6-9a4735e61e4e" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825261 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f326efd-a274-4b02-98f5-b2ca39ca6209" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825268 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="788d8d73-dc49-4bc5-aac2-1a8cbf4e5d10" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825274 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="295069a5-7550-4e03-b1aa-f6c7ae3aa310" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825282 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fcd01ef-3c83-405a-9333-f2bd4d9cf361" containerName="maas-api" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825288 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="089f646b-d23c-481a-b1e3-19fce519cd91" containerName="main" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825293 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa28810-d8d4-4027-b974-fd3b433a46a2" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825300 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="be592e7c-e303-428e-8f0b-906554d7b96b" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825306 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f3562d-4be8-436f-8435-f972c3e98761" containerName="istio-proxy" Oct 29 22:53:18.825584 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.825313 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc290c4e-3d45-4547-bedc-ef18c50947b6" containerName="main" Oct 29 22:53:18.830628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.830605 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.844767 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.844739 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f7a5fe-openshift-default-dockercfg-hvvwh\"" Oct 29 22:53:18.867723 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.867693 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg"] Oct 29 22:53:18.920437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920386 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-data\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.920437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920441 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-socket\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.920654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920533 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-token\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.920654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920594 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-podinfo\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.920654 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920612 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcc4\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-kube-api-access-btcc4\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.920772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920726 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istiod-ca-cert\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.920830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920778 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-certs\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.920830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920802 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-envoy\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.920911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.920894 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-credential-socket\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:18.923157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:18.923130 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:19.039622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.039578 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-podinfo\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.039892 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.039869 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcc4\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-kube-api-access-btcc4\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.040083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.040064 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istiod-ca-cert\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.040206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.040193 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-certs\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.040321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.040306 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-envoy\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.060325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.040535 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-credential-socket\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.060685 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.060655 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-data\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.060836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.060822 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-socket\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.061331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.060915 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-data\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.061555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.056952 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-podinfo\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.061668 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.060667 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-envoy\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.061762 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.046249 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istiod-ca-cert\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.061847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.060987 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-token\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.062046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.046959 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-certs\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.062167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.047822 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-credential-socket\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.062167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.061857 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-socket\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.071510 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.071454 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcc4\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-kube-api-access-btcc4\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.080940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.080859 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-token\") pod \"e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.148563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.148525 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:19.330667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.329495 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:19.480695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.480668 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg"] Oct 29 22:53:19.481812 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:19.481786 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7eef245_d0ec_4b6c_a7e1_2a2c4e806800.slice/crio-a62c28c6971198a757746110a977e6038833e3d25cacf84a63ef458509915ee5 WatchSource:0}: Error finding container a62c28c6971198a757746110a977e6038833e3d25cacf84a63ef458509915ee5: Status 404 returned error can't find the container with id a62c28c6971198a757746110a977e6038833e3d25cacf84a63ef458509915ee5 Oct 29 22:53:19.485071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.485035 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:19.485162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.485103 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:19.485162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.485131 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:19.984188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.984153 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn"] Oct 29 22:53:19.985309 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.985284 5610 cpu_manager.go:414] "containerMap: removing container" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" Oct 29 22:53:19.985437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.985310 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" Oct 29 22:53:19.985558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.985546 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4578473-cc01-462c-920b-c9e76538ca5e" containerName="istio-proxy" Oct 29 22:53:19.990077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:19.990054 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.010671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.010649 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn"] Oct 29 22:53:20.014365 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.014336 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-57d336-openshift-default-dockercfg-5kxjc\"" Oct 29 22:53:20.086097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086065 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872887cc-92d3-405f-b405-6d67d4637f18-istiod-ca-cert\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.086287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086113 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f56x\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-kube-api-access-6f56x\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.086287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086228 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.086287 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086271 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872887cc-92d3-405f-b405-6d67d4637f18-istio-podinfo\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.086508 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086295 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-credential-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.086508 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086457 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-certs\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.086508 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086484 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-istio-token\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.086680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086511 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-envoy\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.086680 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.086539 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-data\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188054 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872887cc-92d3-405f-b405-6d67d4637f18-istio-podinfo\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188097 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-credential-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188224 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-certs\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188253 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-istio-token\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188291 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-envoy\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188313 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-data\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188514 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872887cc-92d3-405f-b405-6d67d4637f18-istiod-ca-cert\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188570 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f56x\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-kube-api-access-6f56x\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.188753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.188641 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.189052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.189027 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.189324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.189299 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-data\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.190021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.189994 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872887cc-92d3-405f-b405-6d67d4637f18-istiod-ca-cert\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.190607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.190580 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-certs\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.190866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.190843 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-credential-socket\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.191925 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.191903 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-envoy\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.196138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.196110 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872887cc-92d3-405f-b405-6d67d4637f18-istio-podinfo\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.201620 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.201588 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" event={"ID":"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800","Type":"ContainerStarted","Data":"41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c"} Oct 29 22:53:20.201734 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.201652 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" event={"ID":"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800","Type":"ContainerStarted","Data":"a62c28c6971198a757746110a977e6038833e3d25cacf84a63ef458509915ee5"} Oct 29 22:53:20.208035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.208003 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-istio-token\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.208424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.208382 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f56x\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-kube-api-access-6f56x\") pod \"e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.240515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.240426 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podStartSLOduration=2.240409232 podStartE2EDuration="2.240409232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:20.238181894 +0000 UTC m=+2262.719499387" watchObservedRunningTime="2026-10-29 22:53:20.240409232 +0000 UTC m=+2262.721726728" Oct 29 22:53:20.274422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.270924 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:20.308476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.307059 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:20.683185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.683109 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn"] Oct 29 22:53:20.687036 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:20.687000 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872887cc_92d3_405f_b405_6d67d4637f18.slice/crio-d6d571236df8882fa285d761350ef993a45f3adfd3eb89b144ea4f6aa8d37b2b WatchSource:0}: Error finding container d6d571236df8882fa285d761350ef993a45f3adfd3eb89b144ea4f6aa8d37b2b: Status 404 returned error can't find the container with id d6d571236df8882fa285d761350ef993a45f3adfd3eb89b144ea4f6aa8d37b2b Oct 29 22:53:20.690802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.690763 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:20.690914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.690857 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:20.690914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:20.690898 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:21.149060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.149027 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:21.155570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.155540 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:21.210341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.210305 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" event={"ID":"872887cc-92d3-405f-b405-6d67d4637f18","Type":"ContainerStarted","Data":"25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966"} Oct 29 22:53:21.210531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.210368 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" event={"ID":"872887cc-92d3-405f-b405-6d67d4637f18","Type":"ContainerStarted","Data":"d6d571236df8882fa285d761350ef993a45f3adfd3eb89b144ea4f6aa8d37b2b"} Oct 29 22:53:21.210788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.210770 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:21.211784 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.211764 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:53:21.253203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.253152 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" podStartSLOduration=2.25313754 podStartE2EDuration="2.25313754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:21.24876396 +0000 UTC m=+2263.730081452" watchObservedRunningTime="2026-10-29 22:53:21.25313754 +0000 UTC m=+2263.734455032" Oct 29 22:53:21.310415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.309756 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:21.320853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.320308 5610 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:53:21.320853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:21.320371 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 29 22:53:22.136445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.131099 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:22.308341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.308301 5610 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:53:22.308574 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.308368 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 29 22:53:22.686433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.686156 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:22.830870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.830764 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:22.847256 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.847227 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:53:22.917675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.917614 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-envoy\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.918132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.917688 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-token\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-token\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.918132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.917735 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-podinfo\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.918433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.918318 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-data\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.918605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.918440 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-kube-api-access-spgz4\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-kube-api-access-spgz4\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.918605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.918496 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-socket\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.918605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.918555 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-credential-socket\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.918816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.918667 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istiod-ca-cert\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.918816 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.918716 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-certs\") pod \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\" (UID: \"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02\") " Oct 29 22:53:22.922224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.919483 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istiod-ca-cert" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:22.922224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.919667 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:22.922224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.919871 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-certs" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.922224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.920737 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-socket" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.922224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.920765 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-podinfo" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:53:22.922224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.921055 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-envoy" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.922224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.922134 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-data" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.922224 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.922203 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-credential-socket" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:22.926429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.925914 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-token" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:22.926429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:22.926134 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-kube-api-access-spgz4" pod "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" (UID: "5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02"). InnerVolumeSpecName "kube-api-access-spgz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:23.022108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.020715 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spgz4\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-kube-api-access-spgz4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.022108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.020750 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.022108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.020766 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.022108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.020780 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.022108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.020794 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.022108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.020807 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.022108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.020820 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.022108 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.020832 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:23.233360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.233320 5610 generic.go:356] "Generic (PLEG): container finished" podID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerID="c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b" exitCode=0 Oct 29 22:53:23.233889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.233362 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" event={"ID":"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02","Type":"ContainerDied","Data":"c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b"} Oct 29 22:53:23.233889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.233433 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" event={"ID":"5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02","Type":"ContainerDied","Data":"17049b1f1254cdca3c942c910bb62d363e04c27e23e3cf2327af80056ea5da62"} Oct 29 22:53:23.233889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.233372 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr" Oct 29 22:53:23.233889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.233461 5610 scope.go:122] "RemoveContainer" containerID="c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b" Oct 29 22:53:23.247118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.247096 5610 scope.go:122] "RemoveContainer" containerID="14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab" Oct 29 22:53:23.299952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.291096 5610 scope.go:122] "RemoveContainer" containerID="c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b" Oct 29 22:53:23.299952 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:23.299676 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b\": container with ID starting with c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b not found: ID does not exist" containerID="c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b" Oct 29 22:53:23.299952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.299722 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b"} err="failed to get container status \"c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b\": rpc error: code = NotFound desc = could not find container \"c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b\": container with ID starting with c9d38ae836e54ce5aac45510daf81fce71354585fb0aaea3d7fc194d320e779b not found: ID does not exist" Oct 29 22:53:23.299952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.299749 5610 scope.go:122] "RemoveContainer" containerID="14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab" Oct 29 22:53:23.302150 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:23.301946 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab\": container with ID starting with 14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab not found: ID does not exist" containerID="14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab" Oct 29 22:53:23.302476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.302297 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab"} err="failed to get container status \"14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab\": rpc error: code = NotFound desc = could not find container \"14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab\": container with ID starting with 14e8ec23b463f202fa3e64158e6e095f6417fa79c8b8a9a43ca756eef153dfab not found: ID does not exist" Oct 29 22:53:23.314573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.314545 5610 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:53:23.316260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.316226 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 29 22:53:23.331495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.331471 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr"] Oct 29 22:53:23.336424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.336387 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ad69b4-openshift-default-b754f7bb9-9nmrr"] Oct 29 22:53:23.637457 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:23.637364 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" path="/var/lib/kubelet/pods/5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02/volumes" Oct 29 22:53:24.312486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.311372 5610 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:53:24.312486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.311461 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 29 22:53:24.337366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.337334 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:24.492418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.491804 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk"] Oct 29 22:53:24.496419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.493078 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" Oct 29 22:53:24.496419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.493105 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" Oct 29 22:53:24.496419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.493162 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" Oct 29 22:53:24.496419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.493171 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" Oct 29 22:53:24.496419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.493348 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" Oct 29 22:53:24.496419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.493367 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc50dc7-12f1-4a23-bbb0-3c76ac0aea02" containerName="istio-proxy" Oct 29 22:53:24.498273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.498236 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:24.503604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.503224 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:24.504032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.504011 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk"] Oct 29 22:53:24.526952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.525894 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7a5fe-dockercfg-l6nht\"" Oct 29 22:53:24.526952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.525951 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7a5fe\"" Oct 29 22:53:24.604477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.593476 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lx7\" (UniqueName: \"kubernetes.io/projected/206056ea-af76-4c20-9333-0fce8991b0c5-kube-api-access-89lx7\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk\" (UID: \"206056ea-af76-4c20-9333-0fce8991b0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:24.604477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.593570 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206056ea-af76-4c20-9333-0fce8991b0c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk\" (UID: \"206056ea-af76-4c20-9333-0fce8991b0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:24.705613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.703123 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lx7\" (UniqueName: \"kubernetes.io/projected/206056ea-af76-4c20-9333-0fce8991b0c5-kube-api-access-89lx7\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk\" (UID: \"206056ea-af76-4c20-9333-0fce8991b0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:24.705613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.703244 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206056ea-af76-4c20-9333-0fce8991b0c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk\" (UID: \"206056ea-af76-4c20-9333-0fce8991b0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:24.705613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.705349 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206056ea-af76-4c20-9333-0fce8991b0c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk\" (UID: \"206056ea-af76-4c20-9333-0fce8991b0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:24.745950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.745907 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk"] Oct 29 22:53:24.825885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.822517 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lx7\" (UniqueName: \"kubernetes.io/projected/206056ea-af76-4c20-9333-0fce8991b0c5-kube-api-access-89lx7\") pod \"payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk\" (UID: \"206056ea-af76-4c20-9333-0fce8991b0c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:24.830406 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.829162 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:24.843422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.840881 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk"] Oct 29 22:53:24.843422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.840983 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:24.920035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.920002 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6prbk\" (UniqueName: \"kubernetes.io/projected/90a6b66a-9f1b-4dde-b410-dba19c24682c-kube-api-access-6prbk\") pod \"payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk\" (UID: \"90a6b66a-9f1b-4dde-b410-dba19c24682c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:24.920194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:24.920059 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a6b66a-9f1b-4dde-b410-dba19c24682c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk\" (UID: \"90a6b66a-9f1b-4dde-b410-dba19c24682c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:25.021741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.021690 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a6b66a-9f1b-4dde-b410-dba19c24682c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk\" (UID: \"90a6b66a-9f1b-4dde-b410-dba19c24682c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:25.022084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.022062 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6prbk\" (UniqueName: \"kubernetes.io/projected/90a6b66a-9f1b-4dde-b410-dba19c24682c-kube-api-access-6prbk\") pod \"payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk\" (UID: \"90a6b66a-9f1b-4dde-b410-dba19c24682c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:25.022712 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.022685 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a6b66a-9f1b-4dde-b410-dba19c24682c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk\" (UID: \"90a6b66a-9f1b-4dde-b410-dba19c24682c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:25.049896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.049850 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6prbk\" (UniqueName: \"kubernetes.io/projected/90a6b66a-9f1b-4dde-b410-dba19c24682c-kube-api-access-6prbk\") pod \"payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk\" (UID: \"90a6b66a-9f1b-4dde-b410-dba19c24682c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:25.204727 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:25.202611 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod206056ea_af76_4c20_9333_0fce8991b0c5.slice/crio-afc3374ff6efa3b989d596fcd8da727c1af0912cc90cdedc082160e10161b9b1 WatchSource:0}: Error finding container afc3374ff6efa3b989d596fcd8da727c1af0912cc90cdedc082160e10161b9b1: Status 404 returned error can't find the container with id afc3374ff6efa3b989d596fcd8da727c1af0912cc90cdedc082160e10161b9b1 Oct 29 22:53:25.208223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.208195 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk"] Oct 29 22:53:25.227947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.227841 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:25.258348 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.258281 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" event={"ID":"206056ea-af76-4c20-9333-0fce8991b0c5","Type":"ContainerStarted","Data":"afc3374ff6efa3b989d596fcd8da727c1af0912cc90cdedc082160e10161b9b1"} Oct 29 22:53:25.307819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.307783 5610 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:53:25.307966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.307844 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 29 22:53:25.594620 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:25.594549 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a6b66a_9f1b_4dde_b410_dba19c24682c.slice/crio-02dc5d3c801751f8b75db7e30a9118a2836a1be75c6ea3a3debbbd48f74810df WatchSource:0}: Error finding container 02dc5d3c801751f8b75db7e30a9118a2836a1be75c6ea3a3debbbd48f74810df: Status 404 returned error can't find the container with id 02dc5d3c801751f8b75db7e30a9118a2836a1be75c6ea3a3debbbd48f74810df Oct 29 22:53:25.596038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:25.596004 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk"] Oct 29 22:53:26.304724 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.304680 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" event={"ID":"206056ea-af76-4c20-9333-0fce8991b0c5","Type":"ContainerStarted","Data":"90cda51ff13cccee6084ac094243f84233ee2a345a61a3adf0c139c1d74ef636"} Oct 29 22:53:26.305407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.305235 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:26.307009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.306949 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" event={"ID":"90a6b66a-9f1b-4dde-b410-dba19c24682c","Type":"ContainerStarted","Data":"fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5"} Oct 29 22:53:26.307135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.307027 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" event={"ID":"90a6b66a-9f1b-4dde-b410-dba19c24682c","Type":"ContainerStarted","Data":"02dc5d3c801751f8b75db7e30a9118a2836a1be75c6ea3a3debbbd48f74810df"} Oct 29 22:53:26.307368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.307351 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:26.307793 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.307769 5610 patch_prober.go:29] interesting pod/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 29 22:53:26.307884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.307831 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 29 22:53:26.350419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.348268 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" podStartSLOduration=2.348247469 podStartE2EDuration="2.348247469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:26.339504535 +0000 UTC m=+2268.820822042" watchObservedRunningTime="2026-10-29 22:53:26.348247469 +0000 UTC m=+2268.829564964" Oct 29 22:53:26.381429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.374100 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:26.401428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:26.398517 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" podStartSLOduration=2.398498336 podStartE2EDuration="2.398498336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:26.393214639 +0000 UTC m=+2268.874532130" watchObservedRunningTime="2026-10-29 22:53:26.398498336 +0000 UTC m=+2268.879815829" Oct 29 22:53:27.457277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:27.457241 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:27.458436 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:27.458373 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:27.460079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:27.460056 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:53:28.039474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:28.035671 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:29.475782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:29.475748 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:31.257632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:31.257536 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:32.426430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:32.422844 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g"] Oct 29 22:53:32.426430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:32.423126 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" podUID="f10315ab-7866-471f-9f67-bece06d8af03" containerName="maas-api" containerID="cri-o://686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0" gracePeriod=30 Oct 29 22:53:32.503272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:32.503238 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:33.030010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.029978 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:53:33.078462 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.078433 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10315ab-7866-471f-9f67-bece06d8af03-kube-api-access-q6gqb\" (UniqueName: \"kubernetes.io/projected/f10315ab-7866-471f-9f67-bece06d8af03-kube-api-access-q6gqb\") pod \"f10315ab-7866-471f-9f67-bece06d8af03\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " Oct 29 22:53:33.078642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.078489 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10315ab-7866-471f-9f67-bece06d8af03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10315ab-7866-471f-9f67-bece06d8af03-openshift-service-ca\") pod \"f10315ab-7866-471f-9f67-bece06d8af03\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " Oct 29 22:53:33.078642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.078537 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-maas-api-tls\") pod \"f10315ab-7866-471f-9f67-bece06d8af03\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " Oct 29 22:53:33.078642 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.078638 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-metrics-certs\") pod \"f10315ab-7866-471f-9f67-bece06d8af03\" (UID: \"f10315ab-7866-471f-9f67-bece06d8af03\") " Oct 29 22:53:33.079374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.078885 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10315ab-7866-471f-9f67-bece06d8af03-openshift-service-ca" pod "f10315ab-7866-471f-9f67-bece06d8af03" (UID: "f10315ab-7866-471f-9f67-bece06d8af03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:53:33.079938 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.079803 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10315ab-7866-471f-9f67-bece06d8af03-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:33.081657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.081630 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10315ab-7866-471f-9f67-bece06d8af03-kube-api-access-q6gqb" pod "f10315ab-7866-471f-9f67-bece06d8af03" (UID: "f10315ab-7866-471f-9f67-bece06d8af03"). InnerVolumeSpecName "kube-api-access-q6gqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:33.081912 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.081888 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-metrics-certs" pod "f10315ab-7866-471f-9f67-bece06d8af03" (UID: "f10315ab-7866-471f-9f67-bece06d8af03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:33.081989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.081964 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-maas-api-tls" pod "f10315ab-7866-471f-9f67-bece06d8af03" (UID: "f10315ab-7866-471f-9f67-bece06d8af03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:33.146306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.146276 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:33.181478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.181408 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6gqb\" (UniqueName: \"kubernetes.io/projected/f10315ab-7866-471f-9f67-bece06d8af03-kube-api-access-q6gqb\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:33.181478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.181436 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:33.181478 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.181450 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10315ab-7866-471f-9f67-bece06d8af03-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:33.303421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.303034 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:33.373052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.373020 5610 generic.go:356] "Generic (PLEG): container finished" podID="f10315ab-7866-471f-9f67-bece06d8af03" containerID="686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0" exitCode=0 Oct 29 22:53:33.373247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.373077 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" Oct 29 22:53:33.373247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.373118 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" event={"ID":"f10315ab-7866-471f-9f67-bece06d8af03","Type":"ContainerDied","Data":"686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0"} Oct 29 22:53:33.373247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.373194 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g" event={"ID":"f10315ab-7866-471f-9f67-bece06d8af03","Type":"ContainerDied","Data":"ae4a2f1c7e5687f4b8a9f465bdf53cb901b1e66784f81ac4aaf1fd6307acd99a"} Oct 29 22:53:33.373247 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.373224 5610 scope.go:122] "RemoveContainer" containerID="686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0" Oct 29 22:53:33.386163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.386138 5610 scope.go:122] "RemoveContainer" containerID="686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0" Oct 29 22:53:33.388456 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:33.386535 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0\": container with ID starting with 686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0 not found: ID does not exist" containerID="686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0" Oct 29 22:53:33.388456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.386574 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0"} err="failed to get container status \"686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0\": rpc error: code = NotFound desc = could not find container \"686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0\": container with ID starting with 686a344c76f5f2439a31e763f3cab808f47a21a69369bfab84be39a2fb0af7b0 not found: ID does not exist" Oct 29 22:53:33.424891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.424860 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g"] Oct 29 22:53:33.428175 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.428152 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8ebc60-69bd864df6-5zl4g"] Oct 29 22:53:33.639340 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.639308 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10315ab-7866-471f-9f67-bece06d8af03" path="/var/lib/kubelet/pods/f10315ab-7866-471f-9f67-bece06d8af03/volumes" Oct 29 22:53:33.833471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:33.833380 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:34.150032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:34.149956 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:34.882077 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:34.882039 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:34.955383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:34.955353 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv"] Oct 29 22:53:35.073639 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:35.073607 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:35.448239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:35.448207 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:35.611054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:35.611021 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:35.811387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:35.811297 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:36.078388 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:36.078300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:37.320844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:37.320809 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:53:37.321644 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:37.321622 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:53:37.382063 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:37.381970 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:39.168607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:39.168569 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:39.674125 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:39.674087 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:40.307427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:40.304941 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:40.467904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:40.467866 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:40.637248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:40.637151 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:40.858965 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:40.858930 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:41.457971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.457935 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:41.819260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.819172 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln"] Oct 29 22:53:41.820285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.820255 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f10315ab-7866-471f-9f67-bece06d8af03" containerName="maas-api" Oct 29 22:53:41.820285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.820283 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10315ab-7866-471f-9f67-bece06d8af03" containerName="maas-api" Oct 29 22:53:41.820548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.820505 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10315ab-7866-471f-9f67-bece06d8af03" containerName="maas-api" Oct 29 22:53:41.824447 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.824420 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.837699 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.837678 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9a15f4-openshift-default-dockercfg-6sn4c\"" Oct 29 22:53:41.852901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.852870 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln"] Oct 29 22:53:41.865276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.865243 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29da9dd4-983c-4343-9439-005194023ce1-istio-podinfo\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.865692 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.865668 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29da9dd4-983c-4343-9439-005194023ce1-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.865910 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.865893 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-istio-token\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.866110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.866090 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77tw\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-kube-api-access-r77tw\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.866301 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.866283 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-certs\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.866451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.866435 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-credential-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.866571 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.866556 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-envoy\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.866695 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.866676 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-data\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.866884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.866863 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.921943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.921909 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:41.972415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.967962 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-certs\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.972415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.968028 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-credential-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.972415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.968064 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-envoy\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.972415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.968092 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-data\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.972415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.968146 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.972415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.968198 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29da9dd4-983c-4343-9439-005194023ce1-istio-podinfo\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.972415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.968347 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29da9dd4-983c-4343-9439-005194023ce1-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.989416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.981243 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.989416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.981548 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-data\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.989416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.981792 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-credential-socket\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:41.989733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:41.989706 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-certs\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.008333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.008299 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-envoy\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.017193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.017158 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29da9dd4-983c-4343-9439-005194023ce1-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.032416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.018297 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-istio-token\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.033142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.033112 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77tw\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-kube-api-access-r77tw\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.034643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.034616 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29da9dd4-983c-4343-9439-005194023ce1-istio-podinfo\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.065420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.060086 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77tw\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-kube-api-access-r77tw\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.065420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.062189 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-istio-token\") pod \"e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.121967 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.121838 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:42.164167 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.164136 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:42.291950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.290630 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:42.440621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.440567 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:42.535952 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:42.535921 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29da9dd4_983c_4343_9439_005194023ce1.slice/crio-7b214a085f9aabfcf28dc4855b67fa762024e1cb624f88caccd1a3ec76f32f21 WatchSource:0}: Error finding container 7b214a085f9aabfcf28dc4855b67fa762024e1cb624f88caccd1a3ec76f32f21: Status 404 returned error can't find the container with id 7b214a085f9aabfcf28dc4855b67fa762024e1cb624f88caccd1a3ec76f32f21 Oct 29 22:53:42.536313 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.536249 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln"] Oct 29 22:53:42.544154 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.544113 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:42.556155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.555528 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:42.556155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.555624 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:42.590181 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.590083 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:42.763417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:42.762870 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:43.431641 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.431608 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:43.464839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.464810 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" event={"ID":"29da9dd4-983c-4343-9439-005194023ce1","Type":"ContainerStarted","Data":"f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec"} Oct 29 22:53:43.465023 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.464860 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" event={"ID":"29da9dd4-983c-4343-9439-005194023ce1","Type":"ContainerStarted","Data":"7b214a085f9aabfcf28dc4855b67fa762024e1cb624f88caccd1a3ec76f32f21"} Oct 29 22:53:43.741965 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:43.741931 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:53:43.808779 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.808588 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podStartSLOduration=2.808520789 podStartE2EDuration="2.808520789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:43.522297487 +0000 UTC m=+2286.003614981" watchObservedRunningTime="2026-10-29 22:53:43.808520789 +0000 UTC m=+2286.289838284" Oct 29 22:53:43.811012 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.810975 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4"] Oct 29 22:53:43.816111 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.816087 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:43.833889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.833818 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-329f7b-openshift-default-dockercfg-fvqhf\"" Oct 29 22:53:43.899366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.899332 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4"] Oct 29 22:53:44.001494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:43.973239 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.019495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.004788 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.019495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.004941 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.019495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.004996 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thd4\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-kube-api-access-7thd4\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.019495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.005054 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.019495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.005097 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.019495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.005160 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.019495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.005202 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.019495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.005235 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.043294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.043254 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:44.108811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.108773 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.110592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.110566 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thd4\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-kube-api-access-7thd4\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.113072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.109441 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.113299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.113267 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.113769 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.113747 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.114311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.114291 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.114994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.114976 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.115491 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.115475 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.115757 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.115737 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.116421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.114718 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.116504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.114192 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.116504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.116332 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.116678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.116663 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.117048 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.116908 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.126726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.126695 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.135298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.135267 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.135845 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.135818 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thd4\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-kube-api-access-7thd4\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.138008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.137906 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.165480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.165442 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:44.174490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.174453 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:44.403505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.403419 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:44.437839 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.437800 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:44.476922 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.476895 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:44.478106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.478084 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:53:44.718966 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.718928 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:44.775677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.775621 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4"] Oct 29 22:53:44.776160 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:44.775826 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f98200_dafd_4f5e_9440_a4e0c37a7b4a.slice/crio-e8e26db2c419ef4d4f0c239805b3d5a6c6be1df86dbc6021ef1360a860860ba0 WatchSource:0}: Error finding container e8e26db2c419ef4d4f0c239805b3d5a6c6be1df86dbc6021ef1360a860860ba0: Status 404 returned error can't find the container with id e8e26db2c419ef4d4f0c239805b3d5a6c6be1df86dbc6021ef1360a860860ba0 Oct 29 22:53:44.779717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.779680 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:44.779812 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.779765 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:44.779856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:44.779809 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:53:45.489810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:45.489769 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" event={"ID":"63f98200-dafd-4f5e-9440-a4e0c37a7b4a","Type":"ContainerStarted","Data":"0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566"} Oct 29 22:53:45.490011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:45.489829 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" event={"ID":"63f98200-dafd-4f5e-9440-a4e0c37a7b4a","Type":"ContainerStarted","Data":"e8e26db2c419ef4d4f0c239805b3d5a6c6be1df86dbc6021ef1360a860860ba0"} Oct 29 22:53:45.532078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:45.532025 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podStartSLOduration=2.532011614 podStartE2EDuration="2.532011614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:45.531213129 +0000 UTC m=+2288.012530622" watchObservedRunningTime="2026-10-29 22:53:45.532011614 +0000 UTC m=+2288.013329106" Oct 29 22:53:45.589245 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:45.589185 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:46.166802 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:46.166739 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:46.356458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:46.356422 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:46.438505 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:46.438422 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:46.444853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:46.444824 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:46.540343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:46.540298 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:46.546283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:46.545166 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:53:46.595871 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:46.595823 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:47.830130 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.830092 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:47.896772 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.896737 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq"] Oct 29 22:53:47.901614 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.901588 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:47.914156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.914129 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-9a15f4\"" Oct 29 22:53:47.927214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.927189 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq"] Oct 29 22:53:47.967830 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.967800 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc"] Oct 29 22:53:47.974126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.974097 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:47.980776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.980744 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd8b6\" (UniqueName: \"kubernetes.io/projected/dcfc3c40-8c05-4bce-885f-424f8742c844-kube-api-access-qd8b6\") pod \"payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq\" (UID: \"dcfc3c40-8c05-4bce-885f-424f8742c844\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:47.980924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.980847 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfc3c40-8c05-4bce-885f-424f8742c844-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq\" (UID: \"dcfc3c40-8c05-4bce-885f-424f8742c844\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:47.990656 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.990560 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-9a15f4-dockercfg-zr5w5\"" Oct 29 22:53:47.993483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.993458 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc"] Oct 29 22:53:47.993984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:47.993948 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:48.135344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.111463 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfc3c40-8c05-4bce-885f-424f8742c844-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq\" (UID: \"dcfc3c40-8c05-4bce-885f-424f8742c844\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:48.135344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.111808 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd8b6\" (UniqueName: \"kubernetes.io/projected/dcfc3c40-8c05-4bce-885f-424f8742c844-kube-api-access-qd8b6\") pod \"payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq\" (UID: \"dcfc3c40-8c05-4bce-885f-424f8742c844\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:48.135344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.113263 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfc3c40-8c05-4bce-885f-424f8742c844-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq\" (UID: \"dcfc3c40-8c05-4bce-885f-424f8742c844\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:48.200416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.194128 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:48.244423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.216416 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e06ae3-0c7d-4158-9e15-63e75c718dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc\" (UID: \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:48.244423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.216490 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5ns\" (UniqueName: \"kubernetes.io/projected/38e06ae3-0c7d-4158-9e15-63e75c718dc0-kube-api-access-mr5ns\") pod \"payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc\" (UID: \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:48.302379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.294015 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd8b6\" (UniqueName: \"kubernetes.io/projected/dcfc3c40-8c05-4bce-885f-424f8742c844-kube-api-access-qd8b6\") pod \"payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq\" (UID: \"dcfc3c40-8c05-4bce-885f-424f8742c844\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:48.320096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.318779 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e06ae3-0c7d-4158-9e15-63e75c718dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc\" (UID: \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:48.320096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.318830 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5ns\" (UniqueName: \"kubernetes.io/projected/38e06ae3-0c7d-4158-9e15-63e75c718dc0-kube-api-access-mr5ns\") pod \"payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc\" (UID: \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:48.320096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.319454 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e06ae3-0c7d-4158-9e15-63e75c718dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc\" (UID: \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:48.336191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.336156 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5ns\" (UniqueName: \"kubernetes.io/projected/38e06ae3-0c7d-4158-9e15-63e75c718dc0-kube-api-access-mr5ns\") pod \"payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc\" (UID: \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:48.348879 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.348848 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:48.518138 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.518104 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:48.610446 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.599753 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:48.803203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.801919 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:48.840445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.840416 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq"] Oct 29 22:53:48.840989 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:48.840966 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfc3c40_8c05_4bce_885f_424f8742c844.slice/crio-3f9b3fd21a852873550cb2213e7d66eb30f2151fb7bfb2639b0752f4c62e5428 WatchSource:0}: Error finding container 3f9b3fd21a852873550cb2213e7d66eb30f2151fb7bfb2639b0752f4c62e5428: Status 404 returned error can't find the container with id 3f9b3fd21a852873550cb2213e7d66eb30f2151fb7bfb2639b0752f4c62e5428 Oct 29 22:53:48.921632 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:48.921605 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc"] Oct 29 22:53:48.921821 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:48.921796 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e06ae3_0c7d_4158_9e15_63e75c718dc0.slice/crio-0c842f06c6336e4d4f18752e70284ed7d5d9ba79825b8a6a82ba10d719bdaeef WatchSource:0}: Error finding container 0c842f06c6336e4d4f18752e70284ed7d5d9ba79825b8a6a82ba10d719bdaeef: Status 404 returned error can't find the container with id 0c842f06c6336e4d4f18752e70284ed7d5d9ba79825b8a6a82ba10d719bdaeef Oct 29 22:53:49.567484 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.567447 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" event={"ID":"dcfc3c40-8c05-4bce-885f-424f8742c844","Type":"ContainerStarted","Data":"d73231b173d83d3699e1153109d31c4d0b54a1be036d6fc2f9e3203cd682cd20"} Oct 29 22:53:49.567686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.567505 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" event={"ID":"dcfc3c40-8c05-4bce-885f-424f8742c844","Type":"ContainerStarted","Data":"3f9b3fd21a852873550cb2213e7d66eb30f2151fb7bfb2639b0752f4c62e5428"} Oct 29 22:53:49.567686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.567535 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:53:49.569345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.569315 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" event={"ID":"38e06ae3-0c7d-4158-9e15-63e75c718dc0","Type":"ContainerStarted","Data":"bb5b33f43de08c384241ad911e0074b8c93c0255fe0c28793bc714b731de794a"} Oct 29 22:53:49.569494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.569360 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" event={"ID":"38e06ae3-0c7d-4158-9e15-63e75c718dc0","Type":"ContainerStarted","Data":"0c842f06c6336e4d4f18752e70284ed7d5d9ba79825b8a6a82ba10d719bdaeef"} Oct 29 22:53:49.569494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.569380 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:53:49.603490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.603425 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" podStartSLOduration=2.603406305 podStartE2EDuration="2.603406305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:49.602066036 +0000 UTC m=+2292.083383543" watchObservedRunningTime="2026-10-29 22:53:49.603406305 +0000 UTC m=+2292.084723792" Oct 29 22:53:49.675088 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.675016 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" podStartSLOduration=2.674995541 podStartE2EDuration="2.674995541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:49.673217263 +0000 UTC m=+2292.154534750" watchObservedRunningTime="2026-10-29 22:53:49.674995541 +0000 UTC m=+2292.156313034" Oct 29 22:53:49.951348 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.951319 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:49.977862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.977826 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" podUID="0e99b243-8803-4980-ab78-cc9c9a1ff7d0" containerName="main" containerID="cri-o://e7452dc946f63b006ce9d127af430e86a71824e49808cf9311508c404bf442be" gracePeriod=45 Oct 29 22:53:49.982961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:49.982919 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" podUID="0e99b243-8803-4980-ab78-cc9c9a1ff7d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.187:8000/ready\": EOF" Oct 29 22:53:50.079546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.079510 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n"] Oct 29 22:53:50.085336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.085311 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:50.106466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.106442 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-329f7b\"" Oct 29 22:53:50.106605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.106463 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-329f7b-dockercfg-dwpr7\"" Oct 29 22:53:50.111232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.111096 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n"] Oct 29 22:53:50.155454 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.155417 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p"] Oct 29 22:53:50.156894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.156862 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:50.158386 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.158339 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng2t9\" (UniqueName: \"kubernetes.io/projected/8566790e-0736-4bbf-845f-3df7bbb82853-kube-api-access-ng2t9\") pod \"payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n\" (UID: \"8566790e-0736-4bbf-845f-3df7bbb82853\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:50.158518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.158471 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8566790e-0736-4bbf-845f-3df7bbb82853-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n\" (UID: \"8566790e-0736-4bbf-845f-3df7bbb82853\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:50.167599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.167574 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:50.257624 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.257592 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p"] Oct 29 22:53:50.297232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.297194 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8566790e-0736-4bbf-845f-3df7bbb82853-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n\" (UID: \"8566790e-0736-4bbf-845f-3df7bbb82853\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:50.311568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.311540 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j675w\" (UniqueName: \"kubernetes.io/projected/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-kube-api-access-j675w\") pod \"payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p\" (UID: \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:50.311944 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.311922 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p\" (UID: \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:50.312254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.312221 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng2t9\" (UniqueName: \"kubernetes.io/projected/8566790e-0736-4bbf-845f-3df7bbb82853-kube-api-access-ng2t9\") pod \"payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n\" (UID: \"8566790e-0736-4bbf-845f-3df7bbb82853\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:50.329555 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.308668 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8566790e-0736-4bbf-845f-3df7bbb82853-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n\" (UID: \"8566790e-0736-4bbf-845f-3df7bbb82853\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:50.439423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.436678 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng2t9\" (UniqueName: \"kubernetes.io/projected/8566790e-0736-4bbf-845f-3df7bbb82853-kube-api-access-ng2t9\") pod \"payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n\" (UID: \"8566790e-0736-4bbf-845f-3df7bbb82853\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:50.439423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.438320 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j675w\" (UniqueName: \"kubernetes.io/projected/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-kube-api-access-j675w\") pod \"payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p\" (UID: \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:50.439423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.438494 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p\" (UID: \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:50.440718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.440691 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p\" (UID: \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:50.469449 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.465926 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j675w\" (UniqueName: \"kubernetes.io/projected/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-kube-api-access-j675w\") pod \"payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p\" (UID: \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:50.579943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.579902 5610 generic.go:356] "Generic (PLEG): container finished" podID="0e99b243-8803-4980-ab78-cc9c9a1ff7d0" containerID="e7452dc946f63b006ce9d127af430e86a71824e49808cf9311508c404bf442be" exitCode=0 Oct 29 22:53:50.580432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.580369 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" event={"ID":"0e99b243-8803-4980-ab78-cc9c9a1ff7d0","Type":"ContainerDied","Data":"e7452dc946f63b006ce9d127af430e86a71824e49808cf9311508c404bf442be"} Oct 29 22:53:50.580515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.580435 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" event={"ID":"0e99b243-8803-4980-ab78-cc9c9a1ff7d0","Type":"ContainerDied","Data":"abe0ad381660a5b4e02bce25a6ce62cee0b1f14a912f319b06640af8facb4c95"} Oct 29 22:53:50.580515 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.580452 5610 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abe0ad381660a5b4e02bce25a6ce62cee0b1f14a912f319b06640af8facb4c95" Oct 29 22:53:50.582142 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.582120 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:53:50.651228 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.651195 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:50.702676 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.702645 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:50.745453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.743772 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-model-cache\") pod \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " Oct 29 22:53:50.745453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.743989 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-dshm\") pod \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " Oct 29 22:53:50.745453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.744057 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tmp-dir\") pod \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " Oct 29 22:53:50.745453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.744091 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tls-certs\") pod \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " Oct 29 22:53:50.745453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.744121 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-home\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-home\") pod \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " Oct 29 22:53:50.745453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.744220 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-kube-api-access-m54gp\" (UniqueName: \"kubernetes.io/projected/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-kube-api-access-m54gp\") pod \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\" (UID: \"0e99b243-8803-4980-ab78-cc9c9a1ff7d0\") " Oct 29 22:53:50.745453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.744716 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tmp-dir" pod "0e99b243-8803-4980-ab78-cc9c9a1ff7d0" (UID: "0e99b243-8803-4980-ab78-cc9c9a1ff7d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:50.745453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.744726 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-model-cache" pod "0e99b243-8803-4980-ab78-cc9c9a1ff7d0" (UID: "0e99b243-8803-4980-ab78-cc9c9a1ff7d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:50.746134 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.746096 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-home" pod "0e99b243-8803-4980-ab78-cc9c9a1ff7d0" (UID: "0e99b243-8803-4980-ab78-cc9c9a1ff7d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:50.749596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.749567 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-kube-api-access-m54gp" pod "0e99b243-8803-4980-ab78-cc9c9a1ff7d0" (UID: "0e99b243-8803-4980-ab78-cc9c9a1ff7d0"). InnerVolumeSpecName "kube-api-access-m54gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:53:50.754187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.754143 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tls-certs" pod "0e99b243-8803-4980-ab78-cc9c9a1ff7d0" (UID: "0e99b243-8803-4980-ab78-cc9c9a1ff7d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:53:50.755428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.755272 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-dshm" pod "0e99b243-8803-4980-ab78-cc9c9a1ff7d0" (UID: "0e99b243-8803-4980-ab78-cc9c9a1ff7d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:53:50.803460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.803019 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:50.851698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.851666 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m54gp\" (UniqueName: \"kubernetes.io/projected/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-kube-api-access-m54gp\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:50.851698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.851698 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:50.851698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.851709 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:50.852008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.851717 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:50.852008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.851724 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:50.852008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.851731 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e99b243-8803-4980-ab78-cc9c9a1ff7d0-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:53:50.967040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:50.967012 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p"] Oct 29 22:53:50.967938 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:50.967913 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b9e7a3_4d2a_4a52_8cc5_da01b5af79ec.slice/crio-459e25ed9fffee6e3f03205f899247c8d6bce86fbbeba1590d2e28ede5e2c829 WatchSource:0}: Error finding container 459e25ed9fffee6e3f03205f899247c8d6bce86fbbeba1590d2e28ede5e2c829: Status 404 returned error can't find the container with id 459e25ed9fffee6e3f03205f899247c8d6bce86fbbeba1590d2e28ede5e2c829 Oct 29 22:53:51.031787 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:53:51.031754 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8566790e_0736_4bbf_845f_3df7bbb82853.slice/crio-cbae3127b6bb490e30727c834ce8196766b78bf1c69d4b1f16c5fcc0062201de WatchSource:0}: Error finding container cbae3127b6bb490e30727c834ce8196766b78bf1c69d4b1f16c5fcc0062201de: Status 404 returned error can't find the container with id cbae3127b6bb490e30727c834ce8196766b78bf1c69d4b1f16c5fcc0062201de Oct 29 22:53:51.032466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.032349 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n"] Oct 29 22:53:51.591172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.591130 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" event={"ID":"8566790e-0736-4bbf-845f-3df7bbb82853","Type":"ContainerStarted","Data":"cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d"} Oct 29 22:53:51.591373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.591202 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" event={"ID":"8566790e-0736-4bbf-845f-3df7bbb82853","Type":"ContainerStarted","Data":"cbae3127b6bb490e30727c834ce8196766b78bf1c69d4b1f16c5fcc0062201de"} Oct 29 22:53:51.591373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.591233 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:53:51.593322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.593300 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv" Oct 29 22:53:51.593477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.593325 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" event={"ID":"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec","Type":"ContainerStarted","Data":"06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378"} Oct 29 22:53:51.593477 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.593371 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" event={"ID":"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec","Type":"ContainerStarted","Data":"459e25ed9fffee6e3f03205f899247c8d6bce86fbbeba1590d2e28ede5e2c829"} Oct 29 22:53:51.593592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.593557 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:53:51.630283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.630203 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" podStartSLOduration=1.630189284 podStartE2EDuration="1.630189284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:51.625860364 +0000 UTC m=+2294.107177857" watchObservedRunningTime="2026-10-29 22:53:51.630189284 +0000 UTC m=+2294.111506774" Oct 29 22:53:51.659049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.659024 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv"] Oct 29 22:53:51.663120 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.663094 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-437981/auth-test-model-437981-kserve-7d6cc8f64f-gbkfv"] Oct 29 22:53:51.694030 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:51.693979 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" podStartSLOduration=1.693963352 podStartE2EDuration="1.693963352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:51.693201772 +0000 UTC m=+2294.174519265" watchObservedRunningTime="2026-10-29 22:53:51.693963352 +0000 UTC m=+2294.175280849" Oct 29 22:53:52.385311 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:52.385274 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:53.637512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:53.637476 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e99b243-8803-4980-ab78-cc9c9a1ff7d0" path="/var/lib/kubelet/pods/0e99b243-8803-4980-ab78-cc9c9a1ff7d0/volumes" Oct 29 22:53:54.121413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:54.121303 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:55.720441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:55.720405 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:57.114616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:57.114576 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:58.672945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.672912 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb"] Oct 29 22:53:58.674778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.674745 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0e99b243-8803-4980-ab78-cc9c9a1ff7d0" containerName="main" Oct 29 22:53:58.674778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.674778 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e99b243-8803-4980-ab78-cc9c9a1ff7d0" containerName="main" Oct 29 22:53:58.674992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.674981 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e99b243-8803-4980-ab78-cc9c9a1ff7d0" containerName="main" Oct 29 22:53:58.681285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.681260 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.700011 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.699984 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb"] Oct 29 22:53:58.705174 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.705147 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f7a5fe-serving-cert\"" Oct 29 22:53:58.775321 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.775286 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.775508 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.775422 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6np\" (UniqueName: \"kubernetes.io/projected/daf75a04-e99c-45c3-a485-611120dda4b0-kube-api-access-7c6np\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.775586 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.775567 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.775718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.775696 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf75a04-e99c-45c3-a485-611120dda4b0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.801364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.801335 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:58.876881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.876841 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.877046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.876998 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6np\" (UniqueName: \"kubernetes.io/projected/daf75a04-e99c-45c3-a485-611120dda4b0-kube-api-access-7c6np\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.877046 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.877037 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.877180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.877109 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf75a04-e99c-45c3-a485-611120dda4b0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.877241 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:58.877178 5610 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-serving-cert: secret "maas-api-e2e-ipp-a-f7a5fe-serving-cert" not found Oct 29 22:53:58.877287 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:53:58.877249 5610 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls podName:daf75a04-e99c-45c3-a485-611120dda4b0 nodeName:}" failed. No retries permitted until 2026-10-29 22:53:59.37722956 +0000 UTC m=+2301.858547048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls") pod "maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" (UID: "daf75a04-e99c-45c3-a485-611120dda4b0") : secret "maas-api-e2e-ipp-a-f7a5fe-serving-cert" not found Oct 29 22:53:58.877807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.877786 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf75a04-e99c-45c3-a485-611120dda4b0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.879338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.879314 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.898121 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.898094 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6np\" (UniqueName: \"kubernetes.io/projected/daf75a04-e99c-45c3-a485-611120dda4b0-kube-api-access-7c6np\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:58.948661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:58.948621 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:59.093163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:59.093127 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:53:59.390383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:59.390281 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:59.393678 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:59.393640 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:59.598866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:59.598832 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:53:59.977412 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:53:59.973999 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb"] Oct 29 22:54:00.010145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:00.010115 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:54:00.428592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:00.428444 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:00.584790 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:00.584766 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 22:54:00.584913 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:00.584847 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 22:54:00.706345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:00.701341 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" event={"ID":"daf75a04-e99c-45c3-a485-611120dda4b0","Type":"ContainerStarted","Data":"350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb"} Oct 29 22:54:00.706345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:00.701422 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" event={"ID":"daf75a04-e99c-45c3-a485-611120dda4b0","Type":"ContainerStarted","Data":"7490783d1da3722b66c9317af6373171804da404c6efaafd377cf5982295fe98"} Oct 29 22:54:00.706345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:00.702468 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:54:00.741901 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:00.741852 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" podStartSLOduration=2.272446812 podStartE2EDuration="2.74183902s" totalImagesPullingTime="469.392208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:00.738427618 +0000 UTC m=+2303.219745113" watchObservedRunningTime="2026-10-29 22:54:00.74183902 +0000 UTC m=+2303.223156512" Oct 29 22:54:02.048417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:02.046189 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:02.605648 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:02.605621 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 22:54:02.606004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:02.605986 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 22:54:03.195429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:03.194016 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:03.913583 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:03.913547 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:05.414569 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:05.410889 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:06.870175 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:06.870018 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:07.724568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:07.724538 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:54:08.413785 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:08.413749 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:10.049607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:10.049570 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:11.608409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:11.608362 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:13.158117 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:13.158049 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:13.714667 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:54:13.714633 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:54:14.879157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:14.879116 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:16.409253 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:16.409212 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:17.927518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:17.927483 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:19.555197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:19.555131 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:20.867931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.867898 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv"] Oct 29 22:54:20.873035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.873015 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:20.887466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.887442 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-9a15f4-serving-cert\"" Oct 29 22:54:20.896040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.896011 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv"] Oct 29 22:54:20.945579 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.945548 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:20.945765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.945675 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tz2x\" (UniqueName: \"kubernetes.io/projected/1ff95e20-7adf-4a33-821d-1d51c58e0d58-kube-api-access-2tz2x\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:20.945765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.945718 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff95e20-7adf-4a33-821d-1d51c58e0d58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:20.945913 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.945891 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:20.996082 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:20.996047 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:21.061827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.061785 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.078826 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.078786 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tz2x\" (UniqueName: \"kubernetes.io/projected/1ff95e20-7adf-4a33-821d-1d51c58e0d58-kube-api-access-2tz2x\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.079506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.079480 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff95e20-7adf-4a33-821d-1d51c58e0d58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.080896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.080679 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff95e20-7adf-4a33-821d-1d51c58e0d58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.081054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.080889 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.095054 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.095022 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.095869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.095843 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.125576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.125502 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tz2x\" (UniqueName: \"kubernetes.io/projected/1ff95e20-7adf-4a33-821d-1d51c58e0d58-kube-api-access-2tz2x\") pod \"maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.191156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.191124 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:21.227501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.227466 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:21.501753 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:54:21.501719 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff95e20_7adf_4a33_821d_1d51c58e0d58.slice/crio-ce4f48145cfb232a520c3f59cc0d0b3af7bf7f048f5ad80b7ad519a1084842f7 WatchSource:0}: Error finding container ce4f48145cfb232a520c3f59cc0d0b3af7bf7f048f5ad80b7ad519a1084842f7: Status 404 returned error can't find the container with id ce4f48145cfb232a520c3f59cc0d0b3af7bf7f048f5ad80b7ad519a1084842f7 Oct 29 22:54:21.502281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.502254 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv"] Oct 29 22:54:21.868604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:21.868573 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" event={"ID":"1ff95e20-7adf-4a33-821d-1d51c58e0d58","Type":"ContainerStarted","Data":"ce4f48145cfb232a520c3f59cc0d0b3af7bf7f048f5ad80b7ad519a1084842f7"} Oct 29 22:54:22.684434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:22.678292 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:22.878278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:22.878237 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" event={"ID":"1ff95e20-7adf-4a33-821d-1d51c58e0d58","Type":"ContainerStarted","Data":"759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75"} Oct 29 22:54:22.878792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:22.878342 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:22.917694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:22.917639 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" podStartSLOduration=2.600106189 podStartE2EDuration="2.91762486s" totalImagesPullingTime="317.518671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:22.915246925 +0000 UTC m=+2325.396564446" watchObservedRunningTime="2026-10-29 22:54:22.91762486 +0000 UTC m=+2325.398942377" Oct 29 22:54:24.305235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:24.305196 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:24.462681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:24.462650 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:24.642283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:24.642164 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:26.005067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:26.004992 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:27.541450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:27.537806 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:28.893094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:28.893064 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 22:54:29.259990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:29.259922 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:30.981417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:30.979914 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:32.534063 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:32.534030 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:34.297663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:34.297622 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:35.645929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:35.645892 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:37.312539 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:37.312414 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:38.355723 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:38.355691 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5"] Oct 29 22:54:38.454791 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:38.454744 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:39.230564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:39.220637 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:39.685102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:39.684990 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:39.965702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:39.965656 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:40.463943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:40.463904 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:40.562939 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:40.562906 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk"] Oct 29 22:54:40.595579 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:40.595537 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk"] Oct 29 22:54:40.765364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:40.765209 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:40.925188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:40.925152 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:41.079998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:41.079906 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:41.272415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:41.261457 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:42.710882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:42.710836 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:43.715476 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:54:43.715428 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:54:44.480364 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:44.480331 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:44.682341 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:44.682307 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:45.564057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:45.563987 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" podUID="90a6b66a-9f1b-4dde-b410-dba19c24682c" containerName="payload-processing" containerID="cri-o://fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5" gracePeriod=25 Oct 29 22:54:45.608417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:45.605497 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" podUID="206056ea-af76-4c20-9333-0fce8991b0c5" containerName="payload-pre-processing" containerID="cri-o://90cda51ff13cccee6084ac094243f84233ee2a345a61a3adf0c139c1d74ef636" gracePeriod=25 Oct 29 22:54:46.259421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.241470 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:46.609469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.581734 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:46.625417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.620546 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:54:46.731443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.727573 5610 generic.go:356] "Generic (PLEG): container finished" podID="90a6b66a-9f1b-4dde-b410-dba19c24682c" containerID="fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5" exitCode=0 Oct 29 22:54:46.731443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.727859 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" event={"ID":"90a6b66a-9f1b-4dde-b410-dba19c24682c","Type":"ContainerDied","Data":"fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5"} Oct 29 22:54:46.731443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.727909 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" event={"ID":"90a6b66a-9f1b-4dde-b410-dba19c24682c","Type":"ContainerDied","Data":"02dc5d3c801751f8b75db7e30a9118a2836a1be75c6ea3a3debbbd48f74810df"} Oct 29 22:54:46.731443 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.727985 5610 scope.go:122] "RemoveContainer" containerID="fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5" Oct 29 22:54:46.781423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.781181 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a6b66a-9f1b-4dde-b410-dba19c24682c-kube-api-access-6prbk\" (UniqueName: \"kubernetes.io/projected/90a6b66a-9f1b-4dde-b410-dba19c24682c-kube-api-access-6prbk\") pod \"90a6b66a-9f1b-4dde-b410-dba19c24682c\" (UID: \"90a6b66a-9f1b-4dde-b410-dba19c24682c\") " Oct 29 22:54:46.781630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.781386 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a6b66a-9f1b-4dde-b410-dba19c24682c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a6b66a-9f1b-4dde-b410-dba19c24682c-plugins-config-volume\") pod \"90a6b66a-9f1b-4dde-b410-dba19c24682c\" (UID: \"90a6b66a-9f1b-4dde-b410-dba19c24682c\") " Oct 29 22:54:46.805418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.784386 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a6b66a-9f1b-4dde-b410-dba19c24682c-plugins-config-volume" pod "90a6b66a-9f1b-4dde-b410-dba19c24682c" (UID: "90a6b66a-9f1b-4dde-b410-dba19c24682c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:46.834800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.803281 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb"] Oct 29 22:54:46.834800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.832957 5610 cpu_manager.go:414] "containerMap: removing container" podUID="90a6b66a-9f1b-4dde-b410-dba19c24682c" containerName="payload-processing" Oct 29 22:54:46.834800 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.832981 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a6b66a-9f1b-4dde-b410-dba19c24682c" containerName="payload-processing" Oct 29 22:54:46.846464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.842910 5610 generic.go:356] "Generic (PLEG): container finished" podID="206056ea-af76-4c20-9333-0fce8991b0c5" containerID="90cda51ff13cccee6084ac094243f84233ee2a345a61a3adf0c139c1d74ef636" exitCode=0 Oct 29 22:54:46.846464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.846093 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a6b66a-9f1b-4dde-b410-dba19c24682c-kube-api-access-6prbk" pod "90a6b66a-9f1b-4dde-b410-dba19c24682c" (UID: "90a6b66a-9f1b-4dde-b410-dba19c24682c"). InnerVolumeSpecName "kube-api-access-6prbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:46.857187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.857146 5610 scope.go:122] "RemoveContainer" containerID="fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5" Oct 29 22:54:46.857683 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:54:46.857660 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5\": container with ID starting with fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5 not found: ID does not exist" containerID="fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5" Oct 29 22:54:46.857862 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.857836 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5"} err="failed to get container status \"fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5\": rpc error: code = NotFound desc = could not find container \"fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5\": container with ID starting with fea93bf36a07d0a292395c6cc6fd554d7c4a6f312257b4c388ae82b06aa881b5 not found: ID does not exist" Oct 29 22:54:46.911143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.910951 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6prbk\" (UniqueName: \"kubernetes.io/projected/90a6b66a-9f1b-4dde-b410-dba19c24682c-kube-api-access-6prbk\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:46.911468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.911446 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90a6b66a-9f1b-4dde-b410-dba19c24682c-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:46.925358 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.925324 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:54:46.938094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.938062 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a6b66a-9f1b-4dde-b410-dba19c24682c" containerName="payload-processing" Oct 29 22:54:46.944374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.944350 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" event={"ID":"206056ea-af76-4c20-9333-0fce8991b0c5","Type":"ContainerDied","Data":"90cda51ff13cccee6084ac094243f84233ee2a345a61a3adf0c139c1d74ef636"} Oct 29 22:54:46.944572 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.944559 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb"] Oct 29 22:54:46.944681 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.944673 5610 scope.go:122] "RemoveContainer" containerID="90cda51ff13cccee6084ac094243f84233ee2a345a61a3adf0c139c1d74ef636" Oct 29 22:54:46.944832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.944794 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:46.958797 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.958779 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09fc89\"" Oct 29 22:54:46.968126 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:46.963543 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09fc89-dockercfg-vqfgb\"" Oct 29 22:54:47.041435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.040941 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/206056ea-af76-4c20-9333-0fce8991b0c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206056ea-af76-4c20-9333-0fce8991b0c5-plugins-config-volume\") pod \"206056ea-af76-4c20-9333-0fce8991b0c5\" (UID: \"206056ea-af76-4c20-9333-0fce8991b0c5\") " Oct 29 22:54:47.041435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.041087 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/206056ea-af76-4c20-9333-0fce8991b0c5-kube-api-access-89lx7\" (UniqueName: \"kubernetes.io/projected/206056ea-af76-4c20-9333-0fce8991b0c5-kube-api-access-89lx7\") pod \"206056ea-af76-4c20-9333-0fce8991b0c5\" (UID: \"206056ea-af76-4c20-9333-0fce8991b0c5\") " Oct 29 22:54:47.042669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.042556 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/206056ea-af76-4c20-9333-0fce8991b0c5-plugins-config-volume" pod "206056ea-af76-4c20-9333-0fce8991b0c5" (UID: "206056ea-af76-4c20-9333-0fce8991b0c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:54:47.042831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.042757 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/206056ea-af76-4c20-9333-0fce8991b0c5-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:47.052623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.051220 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/206056ea-af76-4c20-9333-0fce8991b0c5-kube-api-access-89lx7" pod "206056ea-af76-4c20-9333-0fce8991b0c5" (UID: "206056ea-af76-4c20-9333-0fce8991b0c5"). InnerVolumeSpecName "kube-api-access-89lx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:47.143832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.143785 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb\" (UID: \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:47.144187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.144160 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wftxb\" (UniqueName: \"kubernetes.io/projected/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-kube-api-access-wftxb\") pod \"payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb\" (UID: \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:47.144325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.144309 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lx7\" (UniqueName: \"kubernetes.io/projected/206056ea-af76-4c20-9333-0fce8991b0c5-kube-api-access-89lx7\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:47.235291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.235065 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:47.250451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.248921 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wftxb\" (UniqueName: \"kubernetes.io/projected/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-kube-api-access-wftxb\") pod \"payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb\" (UID: \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:47.250451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.249040 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb\" (UID: \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:47.250451 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.250044 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb\" (UID: \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:47.312532 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.312499 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wftxb\" (UniqueName: \"kubernetes.io/projected/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-kube-api-access-wftxb\") pod \"payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb\" (UID: \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:47.593468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.590170 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:47.929458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.929341 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk" Oct 29 22:54:47.937278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.935565 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" event={"ID":"206056ea-af76-4c20-9333-0fce8991b0c5","Type":"ContainerDied","Data":"afc3374ff6efa3b989d596fcd8da727c1af0912cc90cdedc082160e10161b9b1"} Oct 29 22:54:47.937278 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:47.935708 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk" Oct 29 22:54:48.030078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.029477 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk"] Oct 29 22:54:48.037619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.037597 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a5fe-5c8b67846c-gpwpk"] Oct 29 22:54:48.044856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.044834 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb"] Oct 29 22:54:48.049461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.049436 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk"] Oct 29 22:54:48.052375 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.052315 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-c864778c6-vw2dk"] Oct 29 22:54:48.944828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.944793 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" event={"ID":"d1fb92c8-9ef1-4695-bff5-0f3be79033c1","Type":"ContainerStarted","Data":"7ac6eeae2cd04bb269b859c0d62d0e9821ea039053b9cfd2c87d94c758baf8b8"} Oct 29 22:54:48.945229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.944850 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" event={"ID":"d1fb92c8-9ef1-4695-bff5-0f3be79033c1","Type":"ContainerStarted","Data":"bbcedbd93f02c93e99237e2536c4c47d62e822b2d681accc3b120e396c680cda"} Oct 29 22:54:48.945229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.944896 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:49.023417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:48.985955 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" podStartSLOduration=2.985936216 podStartE2EDuration="2.985936216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:48.983884059 +0000 UTC m=+2351.465201553" watchObservedRunningTime="2026-10-29 22:54:48.985936216 +0000 UTC m=+2351.467253712" Oct 29 22:54:49.025645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:49.025609 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:49.639844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:49.639807 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="206056ea-af76-4c20-9333-0fce8991b0c5" path="/var/lib/kubelet/pods/206056ea-af76-4c20-9333-0fce8991b0c5/volumes" Oct 29 22:54:49.640296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:49.640280 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a6b66a-9f1b-4dde-b410-dba19c24682c" path="/var/lib/kubelet/pods/90a6b66a-9f1b-4dde-b410-dba19c24682c/volumes" Oct 29 22:54:51.073430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:51.071754 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:52.732153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:52.729751 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:53.379266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.379211 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" podUID="4a99d1a9-82e9-4475-a37a-74f69314dad0" containerName="main" containerID="cri-o://2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2" gracePeriod=45 Oct 29 22:54:53.749945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.749917 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:54:53.829483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.827965 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-tmp-dir\") pod \"4a99d1a9-82e9-4475-a37a-74f69314dad0\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " Oct 29 22:54:53.829483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.828016 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-model-cache\") pod \"4a99d1a9-82e9-4475-a37a-74f69314dad0\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " Oct 29 22:54:53.829483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.828300 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-dshm\") pod \"4a99d1a9-82e9-4475-a37a-74f69314dad0\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " Oct 29 22:54:53.829483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.828366 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a99d1a9-82e9-4475-a37a-74f69314dad0-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a99d1a9-82e9-4475-a37a-74f69314dad0-tls-certs\") pod \"4a99d1a9-82e9-4475-a37a-74f69314dad0\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " Oct 29 22:54:53.829483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.828457 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a99d1a9-82e9-4475-a37a-74f69314dad0-kube-api-access-d2cqm\" (UniqueName: \"kubernetes.io/projected/4a99d1a9-82e9-4475-a37a-74f69314dad0-kube-api-access-d2cqm\") pod \"4a99d1a9-82e9-4475-a37a-74f69314dad0\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " Oct 29 22:54:53.829483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.828512 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-home\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-home\") pod \"4a99d1a9-82e9-4475-a37a-74f69314dad0\" (UID: \"4a99d1a9-82e9-4475-a37a-74f69314dad0\") " Oct 29 22:54:53.830414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.830223 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-tmp-dir" pod "4a99d1a9-82e9-4475-a37a-74f69314dad0" (UID: "4a99d1a9-82e9-4475-a37a-74f69314dad0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:53.830913 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.830614 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-model-cache" pod "4a99d1a9-82e9-4475-a37a-74f69314dad0" (UID: "4a99d1a9-82e9-4475-a37a-74f69314dad0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:53.831659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.831362 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-home" pod "4a99d1a9-82e9-4475-a37a-74f69314dad0" (UID: "4a99d1a9-82e9-4475-a37a-74f69314dad0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:53.835044 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.834997 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a99d1a9-82e9-4475-a37a-74f69314dad0-kube-api-access-d2cqm" pod "4a99d1a9-82e9-4475-a37a-74f69314dad0" (UID: "4a99d1a9-82e9-4475-a37a-74f69314dad0"). InnerVolumeSpecName "kube-api-access-d2cqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:54:53.840625 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.840563 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-dshm" pod "4a99d1a9-82e9-4475-a37a-74f69314dad0" (UID: "4a99d1a9-82e9-4475-a37a-74f69314dad0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:54:53.841493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.841435 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a99d1a9-82e9-4475-a37a-74f69314dad0-tls-certs" pod "4a99d1a9-82e9-4475-a37a-74f69314dad0" (UID: "4a99d1a9-82e9-4475-a37a-74f69314dad0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:54:53.930837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.930803 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:53.930837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.930835 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:53.930837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.930843 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:53.931079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.930851 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a99d1a9-82e9-4475-a37a-74f69314dad0-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:53.931079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.930860 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2cqm\" (UniqueName: \"kubernetes.io/projected/4a99d1a9-82e9-4475-a37a-74f69314dad0-kube-api-access-d2cqm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:53.931079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:53.930868 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a99d1a9-82e9-4475-a37a-74f69314dad0-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:54:54.023538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.023449 5610 generic.go:356] "Generic (PLEG): container finished" podID="4a99d1a9-82e9-4475-a37a-74f69314dad0" containerID="2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2" exitCode=0 Oct 29 22:54:54.023538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.023512 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" Oct 29 22:54:54.023730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.023534 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" event={"ID":"4a99d1a9-82e9-4475-a37a-74f69314dad0","Type":"ContainerDied","Data":"2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2"} Oct 29 22:54:54.023730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.023588 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" event={"ID":"4a99d1a9-82e9-4475-a37a-74f69314dad0","Type":"ContainerDied","Data":"4e330ac579296ab743ea0c7d369b7174d9fb09ec18f034e4f03ae3dd86706e23"} Oct 29 22:54:54.023730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.023608 5610 scope.go:122] "RemoveContainer" containerID="2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2" Oct 29 22:54:54.037316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.037293 5610 scope.go:122] "RemoveContainer" containerID="2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2" Oct 29 22:54:54.037708 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:54:54.037683 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2\": container with ID starting with 2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2 not found: ID does not exist" containerID="2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2" Oct 29 22:54:54.037795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.037720 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2"} err="failed to get container status \"2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2\": rpc error: code = NotFound desc = could not find container \"2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2\": container with ID starting with 2efb708556ba8dc72983ae915a6bcad0fe8b51555864e925e0ff5f512b3a7cf2 not found: ID does not exist" Oct 29 22:54:54.071891 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.071863 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5"] Oct 29 22:54:54.079598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.079569 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5"] Oct 29 22:54:54.350193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.350050 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:54.616036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:54.615939 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-45e508/test-model-45e508-kserve-c75c89dc7-t5tc5" podUID="4a99d1a9-82e9-4475-a37a-74f69314dad0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.189:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:54:55.638425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:55.638377 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a99d1a9-82e9-4475-a37a-74f69314dad0" path="/var/lib/kubelet/pods/4a99d1a9-82e9-4475-a37a-74f69314dad0/volumes" Oct 29 22:54:56.187096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.186932 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:56.493065 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.493028 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6"] Oct 29 22:54:56.493957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.493934 5610 cpu_manager.go:414] "containerMap: removing container" podUID="206056ea-af76-4c20-9333-0fce8991b0c5" containerName="payload-pre-processing" Oct 29 22:54:56.493957 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.493956 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="206056ea-af76-4c20-9333-0fce8991b0c5" containerName="payload-pre-processing" Oct 29 22:54:56.494162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.493980 5610 cpu_manager.go:414] "containerMap: removing container" podUID="4a99d1a9-82e9-4475-a37a-74f69314dad0" containerName="main" Oct 29 22:54:56.494162 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.493988 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a99d1a9-82e9-4475-a37a-74f69314dad0" containerName="main" Oct 29 22:54:56.494272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.494228 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="206056ea-af76-4c20-9333-0fce8991b0c5" containerName="payload-pre-processing" Oct 29 22:54:56.494272 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.494249 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a99d1a9-82e9-4475-a37a-74f69314dad0" containerName="main" Oct 29 22:54:56.498044 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.498024 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.510755 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.510726 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-e81b56-openshift-default-dockercfg-cgfcs\"" Oct 29 22:54:56.524345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.524318 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6"] Oct 29 22:54:56.595052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595004 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.595052 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595061 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-data\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.595324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595225 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-certs\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.595324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595286 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-credential-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.595458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595416 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9th\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-kube-api-access-js9th\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.595519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595457 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a39cdad-2f97-46b0-bf8b-9910a2963607-istiod-ca-cert\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.595519 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595482 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-podinfo\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.595633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595551 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-envoy\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.595633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.595600 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-token\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.697673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.697632 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9th\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-kube-api-access-js9th\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.698184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.697717 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a39cdad-2f97-46b0-bf8b-9910a2963607-istiod-ca-cert\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.698184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.697761 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-podinfo\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.698184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.697789 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-envoy\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.698184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.697842 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-token\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.698184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.697983 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.698184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.698006 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-data\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.698184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.698073 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-certs\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.698184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.698126 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-credential-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.700040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.700013 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a39cdad-2f97-46b0-bf8b-9910a2963607-istiod-ca-cert\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.701106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.700754 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.701106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.700881 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-certs\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.701280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.701154 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-data\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.701342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.701312 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-credential-socket\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.703153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.703126 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-podinfo\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.703299 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.703281 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-envoy\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.727949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.727925 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-token\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.729671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.729648 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9th\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-kube-api-access-js9th\") pod \"e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:56.814060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:56.813980 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:57.055708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.055673 5610 generic.go:356] "Generic (PLEG): container finished" podID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerID="41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c" exitCode=0 Oct 29 22:54:57.055951 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.055721 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" event={"ID":"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800","Type":"ContainerDied","Data":"41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c"} Oct 29 22:54:57.056187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.056169 5610 scope.go:122] "RemoveContainer" containerID="41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c" Oct 29 22:54:57.059665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.059627 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:57.059787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.059721 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:57.059787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.059767 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:57.157110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.149552 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6"] Oct 29 22:54:57.158601 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.158557 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:57.158782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.158637 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:57.158782 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:57.158668 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:58.000666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.000626 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:54:58.070889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.070850 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" event={"ID":"9a39cdad-2f97-46b0-bf8b-9910a2963607","Type":"ContainerStarted","Data":"74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308"} Oct 29 22:54:58.071110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.070914 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" event={"ID":"9a39cdad-2f97-46b0-bf8b-9910a2963607","Type":"ContainerStarted","Data":"e360a805a94e05ebed44673a5f12d00a767d012b1f77b4bb1ad034e710966700"} Oct 29 22:54:58.073324 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.073296 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" event={"ID":"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800","Type":"ContainerStarted","Data":"c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e"} Oct 29 22:54:58.075147 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.075122 5610 generic.go:356] "Generic (PLEG): container finished" podID="29da9dd4-983c-4343-9439-005194023ce1" containerID="f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec" exitCode=0 Oct 29 22:54:58.075250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.075161 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" event={"ID":"29da9dd4-983c-4343-9439-005194023ce1","Type":"ContainerDied","Data":"f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec"} Oct 29 22:54:58.075534 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.075518 5610 scope.go:122] "RemoveContainer" containerID="f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec" Oct 29 22:54:58.077633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.077597 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:58.077727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.077688 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:58.077769 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.077736 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:54:58.120612 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.119705 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podStartSLOduration=2.119686582 podStartE2EDuration="2.119686582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:58.109091135 +0000 UTC m=+2360.590408630" watchObservedRunningTime="2026-10-29 22:54:58.119686582 +0000 UTC m=+2360.601004075" Oct 29 22:54:58.157407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.149920 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:54:58.157407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.150043 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 29 22:54:58.157407 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.150086 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 29 22:54:58.827417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.826467 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:54:58.827417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.826597 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:54:58.827417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:58.826641 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 29 22:54:59.132818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.132711 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" event={"ID":"29da9dd4-983c-4343-9439-005194023ce1","Type":"ContainerStarted","Data":"507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad"} Oct 29 22:54:59.149771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.149738 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 29 22:54:59.149937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.149806 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 29 22:54:59.193418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.187572 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:54:59.236662 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.236620 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 29 22:54:59.236982 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.236927 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 29 22:54:59.478418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.477420 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:54:59.822469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.815787 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:54:59.822469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.815850 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 29 22:54:59.956418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.956360 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:54:59.989705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:54:59.989671 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:00.171905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:00.154878 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 29 22:55:00.171905 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:00.154941 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 29 22:55:00.221415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:00.220887 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 29 22:55:00.221415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:00.220945 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 29 22:55:00.820417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:00.819762 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:55:00.820417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:00.819824 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 29 22:55:01.151999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:01.151906 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 29 22:55:01.152267 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:01.152235 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 29 22:55:01.196591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:01.196542 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 29 22:55:01.197183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:01.197134 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 29 22:55:01.839576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:01.839535 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:55:01.841564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:01.841527 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 29 22:55:02.152474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:02.151920 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 29 22:55:02.152474 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:02.152000 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 29 22:55:02.228622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:02.228569 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 29 22:55:02.229282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:02.229224 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 29 22:55:02.588542 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:02.558229 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:02.835414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:02.835243 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:55:02.835414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:02.835318 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 29 22:55:03.005418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.004467 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Oct 29 22:55:03.051840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.050481 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:03.051840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.051749 5610 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.190:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:03.156599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.154723 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 29 22:55:03.156599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.154782 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 29 22:55:03.187421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.186816 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 29 22:55:03.187421 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.186871 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 29 22:55:03.816419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.816115 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:55:03.816419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:03.816179 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 29 22:55:04.169834 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.169744 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 29 22:55:04.170105 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.170078 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 29 22:55:04.248002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.247964 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:55:04.248635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.248609 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:55:04.250040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.250020 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:55:04.559597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.551942 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8"] Oct 29 22:55:04.750374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.750280 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:04.830502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.829715 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 29 22:55:04.830502 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:04.829774 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 29 22:55:05.168872 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:05.168772 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 29 22:55:05.169143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:05.169113 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 29 22:55:05.402414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:05.401938 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:05.820330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:05.820296 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:55:05.820818 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:05.820794 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:55:05.821509 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:05.821482 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 22:55:06.170344 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:06.170265 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:55:06.171468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:06.171447 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 22:55:07.315413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:07.310510 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:07.956040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:07.956006 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:08.399637 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:08.399531 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:08.991904 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:08.991868 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:09.163327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:09.163261 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:09.318490 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:09.312795 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:09.834961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:09.834925 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:10.298663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.298627 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:10.728567 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.728533 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b"] Oct 29 22:55:10.732607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.732583 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.757360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.757335 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e803b6-openshift-default-dockercfg-q4w9g\"" Oct 29 22:55:10.769521 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.769474 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b"] Oct 29 22:55:10.848634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.848575 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.903962 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.904020 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.904057 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.904098 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be13e4-0548-49e6-8c81-006939d66752-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.904158 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.904203 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.904355 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh78m\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-kube-api-access-sh78m\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.904408 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:10.909420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:10.904496 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be13e4-0548-49e6-8c81-006939d66752-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.017843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.017744 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.018132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.018094 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.019235 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.019211 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be13e4-0548-49e6-8c81-006939d66752-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.019501 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.019480 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.019686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.019658 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.026952 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.026930 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-envoy\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.027131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.020969 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be13e4-0548-49e6-8c81-006939d66752-istiod-ca-cert\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.027248 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.021222 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-data\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.031558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.018757 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-certs\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.032214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.032179 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh78m\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-kube-api-access-sh78m\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.032383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.032365 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.036430 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.032613 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be13e4-0548-49e6-8c81-006939d66752-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.037666 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.035008 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-credential-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.037831 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.037615 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.042033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.041864 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-socket\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.053471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.042782 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be13e4-0548-49e6-8c81-006939d66752-istio-podinfo\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.055133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.054212 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:11.064433 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.061093 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-istio-token\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.084168 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.084131 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh78m\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-kube-api-access-sh78m\") pod \"e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.226977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.226941 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:11.347543 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.347463 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:11.672207 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:55:11.672179 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89be13e4_0548_49e6_8c81_006939d66752.slice/crio-00347ebcfcaecc1b94df2133645b8a5faaf0b6d6b43c573bdc9f00f3b17a51f5 WatchSource:0}: Error finding container 00347ebcfcaecc1b94df2133645b8a5faaf0b6d6b43c573bdc9f00f3b17a51f5: Status 404 returned error can't find the container with id 00347ebcfcaecc1b94df2133645b8a5faaf0b6d6b43c573bdc9f00f3b17a51f5 Oct 29 22:55:11.672452 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.672425 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b"] Oct 29 22:55:11.675139 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.675102 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:55:11.675250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.675172 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:55:11.675250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:11.675202 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:55:12.566425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:12.566364 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" event={"ID":"89be13e4-0548-49e6-8c81-006939d66752","Type":"ContainerStarted","Data":"28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9"} Oct 29 22:55:12.566895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:12.566439 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" event={"ID":"89be13e4-0548-49e6-8c81-006939d66752","Type":"ContainerStarted","Data":"00347ebcfcaecc1b94df2133645b8a5faaf0b6d6b43c573bdc9f00f3b17a51f5"} Oct 29 22:55:12.614198 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:12.614152 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" podStartSLOduration=2.614138524 podStartE2EDuration="2.614138524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:12.612824162 +0000 UTC m=+2375.094141655" watchObservedRunningTime="2026-10-29 22:55:12.614138524 +0000 UTC m=+2375.095456090" Oct 29 22:55:12.808225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:12.808166 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:13.046832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:13.046756 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:13.348084 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:13.347999 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:13.354702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:13.354673 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:13.574445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:13.574415 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:13.575419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:13.575368 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:55:13.712924 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:55:13.712891 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:55:14.461721 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.461659 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:14.548796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.548764 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn"] Oct 29 22:55:14.552575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.552559 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:14.571034 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.570996 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-2ae845\"" Oct 29 22:55:14.585867 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.585485 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-2ae845-dockercfg-gr6tr\"" Oct 29 22:55:14.609434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.603535 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn\" (UID: \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:14.609434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.603714 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtfd\" (UniqueName: \"kubernetes.io/projected/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-kube-api-access-9mtfd\") pod \"payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn\" (UID: \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:14.609434 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.604356 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd"] Oct 29 22:55:14.609903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.609881 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:14.610468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.610449 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn"] Oct 29 22:55:14.616869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.616846 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd"] Oct 29 22:55:14.630888 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.630856 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:14.812467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.807468 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8k4t\" (UniqueName: \"kubernetes.io/projected/7cb82a55-445c-40e7-95b3-a40ba0688d71-kube-api-access-z8k4t\") pod \"payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd\" (UID: \"7cb82a55-445c-40e7-95b3-a40ba0688d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:14.840439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.829953 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn\" (UID: \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:14.840439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.830088 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtfd\" (UniqueName: \"kubernetes.io/projected/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-kube-api-access-9mtfd\") pod \"payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn\" (UID: \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:14.840439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.830118 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb82a55-445c-40e7-95b3-a40ba0688d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd\" (UID: \"7cb82a55-445c-40e7-95b3-a40ba0688d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:14.840439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.828253 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:14.853137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.853104 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn\" (UID: \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:14.880286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.880179 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtfd\" (UniqueName: \"kubernetes.io/projected/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-kube-api-access-9mtfd\") pod \"payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn\" (UID: \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:14.897843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.897626 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:14.933528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.932640 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8k4t\" (UniqueName: \"kubernetes.io/projected/7cb82a55-445c-40e7-95b3-a40ba0688d71-kube-api-access-z8k4t\") pod \"payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd\" (UID: \"7cb82a55-445c-40e7-95b3-a40ba0688d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:14.933528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.932807 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb82a55-445c-40e7-95b3-a40ba0688d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd\" (UID: \"7cb82a55-445c-40e7-95b3-a40ba0688d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:14.933841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.933812 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb82a55-445c-40e7-95b3-a40ba0688d71-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd\" (UID: \"7cb82a55-445c-40e7-95b3-a40ba0688d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:14.951439 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:14.951381 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8k4t\" (UniqueName: \"kubernetes.io/projected/7cb82a55-445c-40e7-95b3-a40ba0688d71-kube-api-access-z8k4t\") pod \"payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd\" (UID: \"7cb82a55-445c-40e7-95b3-a40ba0688d71\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:15.216675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.216644 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn"] Oct 29 22:55:15.217779 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:55:15.217749 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cd46c7_31eb_49f4_9ad6_59f83c8f2aed.slice/crio-6aaa07a78be8932eb4116772785121d7ba0e37d595cdaf4cf94d923074535641 WatchSource:0}: Error finding container 6aaa07a78be8932eb4116772785121d7ba0e37d595cdaf4cf94d923074535641: Status 404 returned error can't find the container with id 6aaa07a78be8932eb4116772785121d7ba0e37d595cdaf4cf94d923074535641 Oct 29 22:55:15.227604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.227584 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:15.544367 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.544335 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd"] Oct 29 22:55:15.545931 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:55:15.545903 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb82a55_445c_40e7_95b3_a40ba0688d71.slice/crio-af556123a8ae5f4d8b9b31d754fe559fd4986d313564982d5d4bceace0bf63a8 WatchSource:0}: Error finding container af556123a8ae5f4d8b9b31d754fe559fd4986d313564982d5d4bceace0bf63a8: Status 404 returned error can't find the container with id af556123a8ae5f4d8b9b31d754fe559fd4986d313564982d5d4bceace0bf63a8 Oct 29 22:55:15.602613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.602578 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" event={"ID":"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed","Type":"ContainerStarted","Data":"5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab"} Oct 29 22:55:15.603002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.602643 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" event={"ID":"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed","Type":"ContainerStarted","Data":"6aaa07a78be8932eb4116772785121d7ba0e37d595cdaf4cf94d923074535641"} Oct 29 22:55:15.603002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.602717 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:15.604884 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.604861 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" event={"ID":"7cb82a55-445c-40e7-95b3-a40ba0688d71","Type":"ContainerStarted","Data":"af556123a8ae5f4d8b9b31d754fe559fd4986d313564982d5d4bceace0bf63a8"} Oct 29 22:55:15.605288 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.605216 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:15.640994 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.640943 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" podStartSLOduration=1.640931507 podStartE2EDuration="1.640931507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:15.6395952 +0000 UTC m=+2378.120912695" watchObservedRunningTime="2026-10-29 22:55:15.640931507 +0000 UTC m=+2378.122249000" Oct 29 22:55:15.779222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.779186 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:15.959431 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:15.959257 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:16.615650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:16.615608 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" event={"ID":"7cb82a55-445c-40e7-95b3-a40ba0688d71","Type":"ContainerStarted","Data":"926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee"} Oct 29 22:55:16.616113 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:16.615935 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:16.653440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:16.653371 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" podStartSLOduration=2.653353839 podStartE2EDuration="2.653353839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:16.650690452 +0000 UTC m=+2379.132007945" watchObservedRunningTime="2026-10-29 22:55:16.653353839 +0000 UTC m=+2379.134671333" Oct 29 22:55:17.098128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:17.098090 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:17.448533 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:17.448487 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:17.605236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:17.605192 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:19.071355 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:19.071320 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:19.918886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:19.918845 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerName="main" containerID="cri-o://6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a" gracePeriod=45 Oct 29 22:55:20.043626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.043588 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.190:8000/ready\": dial tcp 10.134.0.190:8000: connect: connection refused" Oct 29 22:55:20.195216 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.195184 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:55:20.280643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.280608 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-dshm\") pod \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " Oct 29 22:55:20.280643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.280654 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-home\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-home\") pod \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " Oct 29 22:55:20.281039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.280972 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-home" pod "bd19e93b-f11d-4cd5-b2ad-bb2740892b87" (UID: "bd19e93b-f11d-4cd5-b2ad-bb2740892b87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:20.281039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.280991 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-model-cache\") pod \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " Oct 29 22:55:20.281225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.281123 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-kube-api-access-zqtw6\" (UniqueName: \"kubernetes.io/projected/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-kube-api-access-zqtw6\") pod \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " Oct 29 22:55:20.281225 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.281217 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tmp-dir\") pod \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " Oct 29 22:55:20.281727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.281251 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tls-certs\") pod \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\" (UID: \"bd19e93b-f11d-4cd5-b2ad-bb2740892b87\") " Oct 29 22:55:20.281727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.281302 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-model-cache" pod "bd19e93b-f11d-4cd5-b2ad-bb2740892b87" (UID: "bd19e93b-f11d-4cd5-b2ad-bb2740892b87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:20.282196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.282087 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:20.282196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.282114 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:20.282618 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.282582 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tmp-dir" pod "bd19e93b-f11d-4cd5-b2ad-bb2740892b87" (UID: "bd19e93b-f11d-4cd5-b2ad-bb2740892b87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:20.284771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.284745 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-dshm" pod "bd19e93b-f11d-4cd5-b2ad-bb2740892b87" (UID: "bd19e93b-f11d-4cd5-b2ad-bb2740892b87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:20.285330 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.285296 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-kube-api-access-zqtw6" pod "bd19e93b-f11d-4cd5-b2ad-bb2740892b87" (UID: "bd19e93b-f11d-4cd5-b2ad-bb2740892b87"). InnerVolumeSpecName "kube-api-access-zqtw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:20.285599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.285568 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tls-certs" pod "bd19e93b-f11d-4cd5-b2ad-bb2740892b87" (UID: "bd19e93b-f11d-4cd5-b2ad-bb2740892b87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:55:20.383546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.383508 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqtw6\" (UniqueName: \"kubernetes.io/projected/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-kube-api-access-zqtw6\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:20.383546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.383539 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:20.383546 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.383556 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:20.383817 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.383567 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd19e93b-f11d-4cd5-b2ad-bb2740892b87-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:20.625426 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.612910 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:20.664308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.663692 5610 generic.go:356] "Generic (PLEG): container finished" podID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerID="6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a" exitCode=0 Oct 29 22:55:20.664308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.663744 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" Oct 29 22:55:20.664308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.663756 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" event={"ID":"bd19e93b-f11d-4cd5-b2ad-bb2740892b87","Type":"ContainerDied","Data":"6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a"} Oct 29 22:55:20.664308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.663797 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8" event={"ID":"bd19e93b-f11d-4cd5-b2ad-bb2740892b87","Type":"ContainerDied","Data":"4aff39aa43509ff24ea590a9cec48a7d1dcadb8c63948bb9687ebff10d67cd4f"} Oct 29 22:55:20.664308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.663819 5610 scope.go:122] "RemoveContainer" containerID="6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a" Oct 29 22:55:20.676570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.676550 5610 scope.go:122] "RemoveContainer" containerID="6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a" Oct 29 22:55:20.676902 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:55:20.676872 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a\": container with ID starting with 6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a not found: ID does not exist" containerID="6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a" Oct 29 22:55:20.677003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.676912 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a"} err="failed to get container status \"6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a\": rpc error: code = NotFound desc = could not find container \"6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a\": container with ID starting with 6f27f96cfbd387a202597f72d212215d74653fbccb56ed8aea13fa8df47df80a not found: ID does not exist" Oct 29 22:55:20.710908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.710879 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8"] Oct 29 22:55:20.720345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:20.720318 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-57d336/test-model-57d336-kserve-85b9c985f9-76sr8"] Oct 29 22:55:21.638659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:21.638624 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" path="/var/lib/kubelet/pods/bd19e93b-f11d-4cd5-b2ad-bb2740892b87/volumes" Oct 29 22:55:21.759320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:21.759284 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:21.916239 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:21.915965 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:22.285513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.285472 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw"] Oct 29 22:55:22.287085 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.287055 5610 cpu_manager.go:414] "containerMap: removing container" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerName="main" Oct 29 22:55:22.289365 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.289344 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerName="main" Oct 29 22:55:22.289962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.289928 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd19e93b-f11d-4cd5-b2ad-bb2740892b87" containerName="main" Oct 29 22:55:22.302607 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.298524 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:22.317829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.317594 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw"] Oct 29 22:55:22.319899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.319836 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7a5fe-dockercfg-zf8tb\"" Oct 29 22:55:22.320730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.320707 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7a5fe\"" Oct 29 22:55:22.338780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.338740 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921b2e-8bdb-47be-ac76-fa83e4607ff4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw\" (UID: \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:22.338978 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.338888 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7sch\" (UniqueName: \"kubernetes.io/projected/31921b2e-8bdb-47be-ac76-fa83e4607ff4-kube-api-access-n7sch\") pod \"payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw\" (UID: \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:22.440760 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.440720 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7sch\" (UniqueName: \"kubernetes.io/projected/31921b2e-8bdb-47be-ac76-fa83e4607ff4-kube-api-access-n7sch\") pod \"payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw\" (UID: \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:22.441187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.441161 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921b2e-8bdb-47be-ac76-fa83e4607ff4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw\" (UID: \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:22.442143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.442115 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921b2e-8bdb-47be-ac76-fa83e4607ff4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw\" (UID: \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:22.470638 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.470606 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7sch\" (UniqueName: \"kubernetes.io/projected/31921b2e-8bdb-47be-ac76-fa83e4607ff4-kube-api-access-n7sch\") pod \"payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw\" (UID: \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:22.627159 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.627070 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:22.824544 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.824499 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:22.934180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:22.934151 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw"] Oct 29 22:55:22.935195 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:55:22.935157 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31921b2e_8bdb_47be_ac76_fa83e4607ff4.slice/crio-4db93bb10514335f3adc2d37b0c03b25b4691003710550a2fd06e86e3ac82861 WatchSource:0}: Error finding container 4db93bb10514335f3adc2d37b0c03b25b4691003710550a2fd06e86e3ac82861: Status 404 returned error can't find the container with id 4db93bb10514335f3adc2d37b0c03b25b4691003710550a2fd06e86e3ac82861 Oct 29 22:55:23.718135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:23.718094 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" event={"ID":"31921b2e-8bdb-47be-ac76-fa83e4607ff4","Type":"ContainerStarted","Data":"4db93bb10514335f3adc2d37b0c03b25b4691003710550a2fd06e86e3ac82861"} Oct 29 22:55:24.694351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:24.689985 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:25.036422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:25.036318 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:25.735458 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:25.735418 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" event={"ID":"31921b2e-8bdb-47be-ac76-fa83e4607ff4","Type":"ContainerStarted","Data":"bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e"} Oct 29 22:55:25.735949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:25.735544 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:25.774647 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:25.774593 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" podStartSLOduration=1.972963616 podStartE2EDuration="3.774578417s" totalImagesPullingTime="1.801614801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:25.770029054 +0000 UTC m=+2388.251346550" watchObservedRunningTime="2026-10-29 22:55:25.774578417 +0000 UTC m=+2388.255895909" Oct 29 22:55:26.460886 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:26.459690 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:26.618955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:26.618924 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 22:55:27.638185 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:27.638151 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 22:55:28.039424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:28.036340 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:29.578056 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:29.578019 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:29.721554 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:29.721512 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:29.867549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:29.857894 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:31.334419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:31.329261 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:32.957298 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:32.957263 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:34.065290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:34.065255 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:34.818789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:34.818751 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:36.381889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:36.381852 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:36.560090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:36.560056 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:36.748176 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:36.748146 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:55:38.209377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:38.209335 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:39.862991 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:39.862949 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:40.497373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:40.497322 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:42.186931 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:42.186890 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:42.425188 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:42.425150 5610 scope.go:122] "RemoveContainer" containerID="e7452dc946f63b006ce9d127af430e86a71824e49808cf9311508c404bf442be" Oct 29 22:55:43.722249 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:55:43.722208 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:55:43.745840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:43.745800 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:45.135932 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:45.135898 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:45.869074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:45.869023 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:46.043246 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:46.040368 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:46.529753 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:46.529709 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:47.582575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:47.582526 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 22:55:47.583106 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:47.582600 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 22:55:47.735873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:47.735826 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:48.136691 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:48.136640 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:48.723282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:48.723249 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:49.235538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:49.235491 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:49.443949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:49.443905 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:49.774258 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:49.774089 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:50.086722 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:50.086639 5610 generic.go:356] "Generic (PLEG): container finished" podID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerID="0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566" exitCode=0 Oct 29 22:55:50.086894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:50.086718 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" event={"ID":"63f98200-dafd-4f5e-9440-a4e0c37a7b4a","Type":"ContainerDied","Data":"0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566"} Oct 29 22:55:50.087283 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:50.087262 5610 scope.go:122] "RemoveContainer" containerID="0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566" Oct 29 22:55:50.089205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:50.089175 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:55:50.089300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:50.089249 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:55:50.089300 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:50.089283 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:55:51.106876 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:51.106831 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" event={"ID":"63f98200-dafd-4f5e-9440-a4e0c37a7b4a","Type":"ContainerStarted","Data":"0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb"} Oct 29 22:55:51.324659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:51.324617 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:51.438566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:51.438478 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:55:51.438742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:51.438576 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:55:51.438742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:51.438608 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:55:52.354119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:52.354085 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:52.438442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:52.438386 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:55:52.438619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:52.438486 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:55:52.521611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:52.521553 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:52.782723 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:52.782685 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:53.011997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:53.011952 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:53.318480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:53.306945 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:53.323590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:53.323553 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk"] Oct 29 22:55:53.323881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:53.323851 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" containerID="cri-o://4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4" gracePeriod=30 Oct 29 22:55:53.458914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:53.458863 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:53.472670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:53.472632 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:55:53.472947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:53.472919 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:55:54.060556 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:54.058329 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:54.438766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:54.438677 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:55:54.438766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:54.438747 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:55:55.058672 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.058637 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:55.129530 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.129496 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t"] Oct 29 22:55:55.135728 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.135706 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:55.148796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.148774 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6f83c\"" Oct 29 22:55:55.158907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.158878 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6f83c-dockercfg-92bfg\"" Oct 29 22:55:55.193422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.191781 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t"] Oct 29 22:55:55.226335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.226290 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5e0990-dc13-4f97-97db-f21eec274ff6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t\" (UID: \"9f5e0990-dc13-4f97-97db-f21eec274ff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:55.226335 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.226310 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:55.226599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.226488 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdzd\" (UniqueName: \"kubernetes.io/projected/9f5e0990-dc13-4f97-97db-f21eec274ff6-kube-api-access-jzdzd\") pod \"payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t\" (UID: \"9f5e0990-dc13-4f97-97db-f21eec274ff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:55.358459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.354211 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdzd\" (UniqueName: \"kubernetes.io/projected/9f5e0990-dc13-4f97-97db-f21eec274ff6-kube-api-access-jzdzd\") pod \"payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t\" (UID: \"9f5e0990-dc13-4f97-97db-f21eec274ff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:55.414503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.414463 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5e0990-dc13-4f97-97db-f21eec274ff6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t\" (UID: \"9f5e0990-dc13-4f97-97db-f21eec274ff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:55.419415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.415696 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5e0990-dc13-4f97-97db-f21eec274ff6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t\" (UID: \"9f5e0990-dc13-4f97-97db-f21eec274ff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:55.455415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.455200 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:55:55.455415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.455255 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:55:55.500059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.500023 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdzd\" (UniqueName: \"kubernetes.io/projected/9f5e0990-dc13-4f97-97db-f21eec274ff6-kube-api-access-jzdzd\") pod \"payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t\" (UID: \"9f5e0990-dc13-4f97-97db-f21eec274ff6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:55.750798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:55.750765 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:56.099977 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:56.099886 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t"] Oct 29 22:55:56.112530 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:55:56.112484 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5e0990_dc13_4f97_97db_f21eec274ff6.slice/crio-b8313b40b02b171274e1436bc20128858f889264c33122d36b6cd87f234779e9 WatchSource:0}: Error finding container b8313b40b02b171274e1436bc20128858f889264c33122d36b6cd87f234779e9: Status 404 returned error can't find the container with id b8313b40b02b171274e1436bc20128858f889264c33122d36b6cd87f234779e9 Oct 29 22:55:56.345534 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:56.344470 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" event={"ID":"9f5e0990-dc13-4f97-97db-f21eec274ff6","Type":"ContainerStarted","Data":"b8313b40b02b171274e1436bc20128858f889264c33122d36b6cd87f234779e9"} Oct 29 22:55:56.381688 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:56.381195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:56.449257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:56.441776 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body= Oct 29 22:55:56.449257 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:56.441860 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" Oct 29 22:55:56.532985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:56.532936 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:57.370326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:57.370280 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" event={"ID":"9f5e0990-dc13-4f97-97db-f21eec274ff6","Type":"ContainerStarted","Data":"3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc"} Oct 29 22:55:57.371326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:57.371299 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:55:57.412051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:57.411989 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" podStartSLOduration=2.41197188 podStartE2EDuration="2.41197188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:57.407165732 +0000 UTC m=+2419.888483225" watchObservedRunningTime="2026-10-29 22:55:57.41197188 +0000 UTC m=+2419.893289374" Oct 29 22:55:57.443169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:57.443136 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:55:57.443520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:57.443495 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:55:57.444114 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:57.444093 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 22:55:58.185417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.180084 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:58.638427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.638385 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:55:58.701856 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.701820 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-envoy\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.702025 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.701875 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-credential-socket\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.702025 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.701966 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-socket\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.702118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.702038 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-token\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-token\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.702163 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.702134 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-certs\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.702201 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.702177 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-data\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.702238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.702223 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-kube-api-access-t4wd8\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-kube-api-access-t4wd8\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.702357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.702342 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istiod-ca-cert\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.702497 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.702483 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-podinfo\") pod \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\" (UID: \"6f82bd6f-638c-45f1-8bd0-2d7d402c881b\") " Oct 29 22:55:58.705031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.704105 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-credential-socket" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.705031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.704145 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istiod-ca-cert" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:55:58.705031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.704305 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-socket" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.705031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.704419 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-certs" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.705031 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.704641 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-data" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.707197 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.707167 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-token" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:58.709980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.709859 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-podinfo" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:55:58.709980 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.709957 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-kube-api-access-t4wd8" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "kube-api-access-t4wd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:55:58.720893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.720863 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-envoy" pod "6f82bd6f-638c-45f1-8bd0-2d7d402c881b" (UID: "6f82bd6f-638c-45f1-8bd0-2d7d402c881b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:55:58.805455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805416 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wd8\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-kube-api-access-t4wd8\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.805455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805457 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.805650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805472 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.805650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805484 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.805650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805500 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.805650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805513 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.805650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805525 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.805650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805537 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:58.805650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:58.805548 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f82bd6f-638c-45f1-8bd0-2d7d402c881b-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:55:59.391409 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.391352 5610 generic.go:356] "Generic (PLEG): container finished" podID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerID="4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4" exitCode=0 Oct 29 22:55:59.391623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.391561 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" Oct 29 22:55:59.391623 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.391593 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" event={"ID":"6f82bd6f-638c-45f1-8bd0-2d7d402c881b","Type":"ContainerDied","Data":"4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4"} Oct 29 22:55:59.391745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.391661 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk" event={"ID":"6f82bd6f-638c-45f1-8bd0-2d7d402c881b","Type":"ContainerDied","Data":"4504ea6bcea1a3ecb60be0f668786ee1ddd809bff3c94a74659c71087e87299d"} Oct 29 22:55:59.391745 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.391693 5610 scope.go:122] "RemoveContainer" containerID="4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4" Oct 29 22:55:59.405726 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.405706 5610 scope.go:122] "RemoveContainer" containerID="4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4" Oct 29 22:55:59.406076 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:55:59.406052 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4\": container with ID starting with 4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4 not found: ID does not exist" containerID="4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4" Oct 29 22:55:59.406153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.406086 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4"} err="failed to get container status \"4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4\": rpc error: code = NotFound desc = could not find container \"4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4\": container with ID starting with 4e28698ddeb22610035a1cf83070450b48bf139ab78321b99dc8dc68f53c51c4 not found: ID does not exist" Oct 29 22:55:59.413173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.413151 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:55:59.456881 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.456811 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk"] Oct 29 22:55:59.460580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.460553 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-45e508-openshift-default-585ccbd46f-4m7zk"] Oct 29 22:55:59.637970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.637936 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" path="/var/lib/kubelet/pods/6f82bd6f-638c-45f1-8bd0-2d7d402c881b/volumes" Oct 29 22:55:59.948686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.948650 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4"] Oct 29 22:55:59.950010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.949985 5610 cpu_manager.go:414] "containerMap: removing container" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" Oct 29 22:55:59.950010 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.950008 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" Oct 29 22:55:59.950238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.950194 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f82bd6f-638c-45f1-8bd0-2d7d402c881b" containerName="istio-proxy" Oct 29 22:55:59.957448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.957421 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:55:59.979671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.979646 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-e803b6-serving-cert\"" Oct 29 22:55:59.991807 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:55:59.991779 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4"] Oct 29 22:56:00.024136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.024103 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.024318 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.024156 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.024463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.024371 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ll5r\" (UniqueName: \"kubernetes.io/projected/f91681c3-825b-4be3-a29a-6231db8b969a-kube-api-access-8ll5r\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.024562 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.024540 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91681c3-825b-4be3-a29a-6231db8b969a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.105255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.105212 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:00.126158 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.126128 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.126308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.126184 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.126308 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.126294 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ll5r\" (UniqueName: \"kubernetes.io/projected/f91681c3-825b-4be3-a29a-6231db8b969a-kube-api-access-8ll5r\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.126425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.126362 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91681c3-825b-4be3-a29a-6231db8b969a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.126998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.126971 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91681c3-825b-4be3-a29a-6231db8b969a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.128810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.128785 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.128810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.128801 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.145199 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.145171 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ll5r\" (UniqueName: \"kubernetes.io/projected/f91681c3-825b-4be3-a29a-6231db8b969a-kube-api-access-8ll5r\") pod \"maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.275643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.275557 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:00.322498 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.322468 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:00.557720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.557644 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:00.594973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.594943 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4"] Oct 29 22:56:00.595628 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:56:00.595603 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91681c3_825b_4be3_a29a_6231db8b969a.slice/crio-d90bbe44dd199241e2ac3cb37dde5b5ba41181dcccc6833e1f0fb7471522e0b2 WatchSource:0}: Error finding container d90bbe44dd199241e2ac3cb37dde5b5ba41181dcccc6833e1f0fb7471522e0b2: Status 404 returned error can't find the container with id d90bbe44dd199241e2ac3cb37dde5b5ba41181dcccc6833e1f0fb7471522e0b2 Oct 29 22:56:00.750789 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:00.750737 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:01.411740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:01.411645 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" event={"ID":"f91681c3-825b-4be3-a29a-6231db8b969a","Type":"ContainerStarted","Data":"a60e0bd80a011f02390210ba649e4d2e3b496b1968a944544805eca3346f9b86"} Oct 29 22:56:01.411740 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:01.411713 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" event={"ID":"f91681c3-825b-4be3-a29a-6231db8b969a","Type":"ContainerStarted","Data":"d90bbe44dd199241e2ac3cb37dde5b5ba41181dcccc6833e1f0fb7471522e0b2"} Oct 29 22:56:01.412156 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:01.411870 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:01.451730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:01.451678 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" podStartSLOduration=1.9913757319999998 podStartE2EDuration="2.451665474s" totalImagesPullingTime="460.289742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:01.449106803 +0000 UTC m=+2423.930424297" watchObservedRunningTime="2026-10-29 22:56:01.451665474 +0000 UTC m=+2423.932982966" Oct 29 22:56:01.911347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:01.911316 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:03.463416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:03.458826 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:04.615682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:04.615632 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:04.617468 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:04.617447 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw"] Oct 29 22:56:04.669137 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:04.669100 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8"] Oct 29 22:56:04.801096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:04.800988 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:04.957667 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:04.957492 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:05.448223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:05.448173 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:06.792969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:06.792934 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:06.943376 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:06.943336 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:07.425180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:07.425151 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 22:56:07.538302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:07.538273 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:09.396598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:09.396571 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:56:09.618151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:09.618109 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" podUID="8191db7b-cb6a-4aae-b00c-c2232aa8f278" containerName="payload-processing" containerID="cri-o://d24c998fd0621864d3f7f016481efff8b537dd70b13a0fe27152dafe2d61d902" gracePeriod=25 Oct 29 22:56:09.669853 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:09.669751 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" podUID="c25ff675-008c-46fa-808f-ea1bf5a30670" containerName="payload-pre-processing" containerID="cri-o://c766bf93ba3e6ca8c824e653da10efdd51997807e0ed6d1db76bb7f5926290f5" gracePeriod=25 Oct 29 22:56:10.015815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:10.015777 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:11.758718 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:11.758672 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:12.706047 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:12.706001 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" podUID="8191db7b-cb6a-4aae-b00c-c2232aa8f278" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:56:12.706677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:12.706643 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" podUID="c25ff675-008c-46fa-808f-ea1bf5a30670" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:56:13.621900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:13.621829 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:13.725701 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:56:13.725657 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:56:15.224882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:15.224832 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:16.704437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:16.703802 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:18.381059 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:18.380985 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:20.228302 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:20.228172 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:21.493835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:21.493799 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb"] Oct 29 22:56:21.680128 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:21.680087 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:21.814634 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:21.814538 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:21.968683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:21.968629 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:22.681761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:22.681708 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" podUID="c25ff675-008c-46fa-808f-ea1bf5a30670" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:56:22.682356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:22.681792 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" podUID="8191db7b-cb6a-4aae-b00c-c2232aa8f278" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:56:23.293995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:23.293957 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:24.854422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:24.853519 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:26.283327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.283276 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:26.494751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.494703 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" podUID="d1fb92c8-9ef1-4695-bff5-0f3be79033c1" containerName="payload-processing" containerID="cri-o://7ac6eeae2cd04bb269b859c0d62d0e9821ea039053b9cfd2c87d94c758baf8b8" gracePeriod=25 Oct 29 22:56:26.805790 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.805760 5610 generic.go:356] "Generic (PLEG): container finished" podID="d1fb92c8-9ef1-4695-bff5-0f3be79033c1" containerID="7ac6eeae2cd04bb269b859c0d62d0e9821ea039053b9cfd2c87d94c758baf8b8" exitCode=0 Oct 29 22:56:26.805960 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.805834 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" event={"ID":"d1fb92c8-9ef1-4695-bff5-0f3be79033c1","Type":"ContainerDied","Data":"7ac6eeae2cd04bb269b859c0d62d0e9821ea039053b9cfd2c87d94c758baf8b8"} Oct 29 22:56:26.805960 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.805892 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" event={"ID":"d1fb92c8-9ef1-4695-bff5-0f3be79033c1","Type":"ContainerDied","Data":"bbcedbd93f02c93e99237e2536c4c47d62e822b2d681accc3b120e396c680cda"} Oct 29 22:56:26.805960 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.805917 5610 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbcedbd93f02c93e99237e2536c4c47d62e822b2d681accc3b120e396c680cda" Oct 29 22:56:26.811464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.811443 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:56:26.903296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.903200 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-plugins-config-volume\") pod \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\" (UID: \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\") " Oct 29 22:56:26.903495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.903443 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-kube-api-access-wftxb\" (UniqueName: \"kubernetes.io/projected/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-kube-api-access-wftxb\") pod \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\" (UID: \"d1fb92c8-9ef1-4695-bff5-0f3be79033c1\") " Oct 29 22:56:26.903696 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.903668 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-plugins-config-volume" pod "d1fb92c8-9ef1-4695-bff5-0f3be79033c1" (UID: "d1fb92c8-9ef1-4695-bff5-0f3be79033c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:26.904119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.904095 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:26.906366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:26.906326 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-kube-api-access-wftxb" pod "d1fb92c8-9ef1-4695-bff5-0f3be79033c1" (UID: "d1fb92c8-9ef1-4695-bff5-0f3be79033c1"). InnerVolumeSpecName "kube-api-access-wftxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:27.005379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:27.005337 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wftxb\" (UniqueName: \"kubernetes.io/projected/d1fb92c8-9ef1-4695-bff5-0f3be79033c1-kube-api-access-wftxb\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:27.786962 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:27.786898 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:27.845990 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:27.845958 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb" Oct 29 22:56:27.893066 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:27.893035 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb"] Oct 29 22:56:27.902583 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:27.902512 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-695597ffd8-bntdb"] Oct 29 22:56:29.212606 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:29.212566 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:29.406296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:29.406259 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:29.639861 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:29.639832 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fb92c8-9ef1-4695-bff5-0f3be79033c1" path="/var/lib/kubelet/pods/d1fb92c8-9ef1-4695-bff5-0f3be79033c1/volumes" Oct 29 22:56:29.877997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:29.877896 5610 generic.go:356] "Generic (PLEG): container finished" podID="8191db7b-cb6a-4aae-b00c-c2232aa8f278" containerID="d24c998fd0621864d3f7f016481efff8b537dd70b13a0fe27152dafe2d61d902" exitCode=0 Oct 29 22:56:29.878312 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:29.878031 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" event={"ID":"8191db7b-cb6a-4aae-b00c-c2232aa8f278","Type":"ContainerDied","Data":"d24c998fd0621864d3f7f016481efff8b537dd70b13a0fe27152dafe2d61d902"} Oct 29 22:56:29.880617 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:29.880577 5610 generic.go:356] "Generic (PLEG): container finished" podID="c25ff675-008c-46fa-808f-ea1bf5a30670" containerID="c766bf93ba3e6ca8c824e653da10efdd51997807e0ed6d1db76bb7f5926290f5" exitCode=0 Oct 29 22:56:29.880840 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:29.880815 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" event={"ID":"c25ff675-008c-46fa-808f-ea1bf5a30670","Type":"ContainerDied","Data":"c766bf93ba3e6ca8c824e653da10efdd51997807e0ed6d1db76bb7f5926290f5"} Oct 29 22:56:29.991900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:29.991871 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:56:30.024151 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.024128 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:56:30.053651 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.053619 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8191db7b-cb6a-4aae-b00c-c2232aa8f278-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8191db7b-cb6a-4aae-b00c-c2232aa8f278-plugins-config-volume\") pod \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\" (UID: \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\") " Oct 29 22:56:30.053819 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.053768 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8191db7b-cb6a-4aae-b00c-c2232aa8f278-kube-api-access-bzwmv\" (UniqueName: \"kubernetes.io/projected/8191db7b-cb6a-4aae-b00c-c2232aa8f278-kube-api-access-bzwmv\") pod \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\" (UID: \"8191db7b-cb6a-4aae-b00c-c2232aa8f278\") " Oct 29 22:56:30.053895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.053830 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25ff675-008c-46fa-808f-ea1bf5a30670-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ff675-008c-46fa-808f-ea1bf5a30670-plugins-config-volume\") pod \"c25ff675-008c-46fa-808f-ea1bf5a30670\" (UID: \"c25ff675-008c-46fa-808f-ea1bf5a30670\") " Oct 29 22:56:30.054014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.053998 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25ff675-008c-46fa-808f-ea1bf5a30670-kube-api-access-z88pd\" (UniqueName: \"kubernetes.io/projected/c25ff675-008c-46fa-808f-ea1bf5a30670-kube-api-access-z88pd\") pod \"c25ff675-008c-46fa-808f-ea1bf5a30670\" (UID: \"c25ff675-008c-46fa-808f-ea1bf5a30670\") " Oct 29 22:56:30.054186 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.054160 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8191db7b-cb6a-4aae-b00c-c2232aa8f278-plugins-config-volume" pod "8191db7b-cb6a-4aae-b00c-c2232aa8f278" (UID: "8191db7b-cb6a-4aae-b00c-c2232aa8f278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:30.054261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.054222 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25ff675-008c-46fa-808f-ea1bf5a30670-plugins-config-volume" pod "c25ff675-008c-46fa-808f-ea1bf5a30670" (UID: "c25ff675-008c-46fa-808f-ea1bf5a30670"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:30.054729 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.054702 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8191db7b-cb6a-4aae-b00c-c2232aa8f278-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:30.054852 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.054731 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25ff675-008c-46fa-808f-ea1bf5a30670-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:30.056783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.056757 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8191db7b-cb6a-4aae-b00c-c2232aa8f278-kube-api-access-bzwmv" pod "8191db7b-cb6a-4aae-b00c-c2232aa8f278" (UID: "8191db7b-cb6a-4aae-b00c-c2232aa8f278"). InnerVolumeSpecName "kube-api-access-bzwmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:30.056895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.056854 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25ff675-008c-46fa-808f-ea1bf5a30670-kube-api-access-z88pd" pod "c25ff675-008c-46fa-808f-ea1bf5a30670" (UID: "c25ff675-008c-46fa-808f-ea1bf5a30670"). InnerVolumeSpecName "kube-api-access-z88pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:30.156022 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.155934 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzwmv\" (UniqueName: \"kubernetes.io/projected/8191db7b-cb6a-4aae-b00c-c2232aa8f278-kube-api-access-bzwmv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:30.156022 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.155969 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88pd\" (UniqueName: \"kubernetes.io/projected/c25ff675-008c-46fa-808f-ea1bf5a30670-kube-api-access-z88pd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:30.905899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.905814 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" Oct 29 22:56:30.905899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.905817 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw" event={"ID":"8191db7b-cb6a-4aae-b00c-c2232aa8f278","Type":"ContainerDied","Data":"8f56a3cadd7e3251a165021a5be8c5f00632f68e4ff76c2d18e245d2eb09099c"} Oct 29 22:56:30.906444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.905926 5610 scope.go:122] "RemoveContainer" containerID="d24c998fd0621864d3f7f016481efff8b537dd70b13a0fe27152dafe2d61d902" Oct 29 22:56:30.908075 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.908044 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" event={"ID":"c25ff675-008c-46fa-808f-ea1bf5a30670","Type":"ContainerDied","Data":"03a71992b5d053ff2b1352caf545299feff86cfccc1d80d08d804554d86fe05d"} Oct 29 22:56:30.908382 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.908365 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8" Oct 29 22:56:30.934440 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.919128 5610 scope.go:122] "RemoveContainer" containerID="c766bf93ba3e6ca8c824e653da10efdd51997807e0ed6d1db76bb7f5926290f5" Oct 29 22:56:30.954580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:30.954541 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:31.016244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:31.016212 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8"] Oct 29 22:56:31.025859 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:31.025832 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-57d336-6f75d45b47-bsrt8"] Oct 29 22:56:31.050014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:31.049987 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw"] Oct 29 22:56:31.054943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:31.054919 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-57d336-5f55d66b86-xqdhw"] Oct 29 22:56:31.639072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:31.639031 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8191db7b-cb6a-4aae-b00c-c2232aa8f278" path="/var/lib/kubelet/pods/8191db7b-cb6a-4aae-b00c-c2232aa8f278/volumes" Oct 29 22:56:31.639513 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:31.639495 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25ff675-008c-46fa-808f-ea1bf5a30670" path="/var/lib/kubelet/pods/c25ff675-008c-46fa-808f-ea1bf5a30670/volumes" Oct 29 22:56:32.503418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:32.500793 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:32.641835 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:32.641801 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:33.949166 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:33.949131 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:35.657456 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:35.657386 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:35.803078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:35.803022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:37.348074 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:37.348017 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:38.641893 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:38.641858 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr"] Oct 29 22:56:38.642323 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:38.642038 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" podUID="84a7170c-6579-469d-a4ed-deb08c89a86f" containerName="maas-api" containerID="cri-o://498dd05a0729b38bdcc29a0fde9edc6a4284b00f1263dbb574882ed565511d82" gracePeriod=30 Oct 29 22:56:38.755293 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:38.755263 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:39.148429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.136498 5610 generic.go:356] "Generic (PLEG): container finished" podID="84a7170c-6579-469d-a4ed-deb08c89a86f" containerID="498dd05a0729b38bdcc29a0fde9edc6a4284b00f1263dbb574882ed565511d82" exitCode=0 Oct 29 22:56:39.148429 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.136716 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" event={"ID":"84a7170c-6579-469d-a4ed-deb08c89a86f","Type":"ContainerDied","Data":"498dd05a0729b38bdcc29a0fde9edc6a4284b00f1263dbb574882ed565511d82"} Oct 29 22:56:39.257254 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.257180 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:56:39.340918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.340886 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-maas-api-tls\") pod \"84a7170c-6579-469d-a4ed-deb08c89a86f\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " Oct 29 22:56:39.341122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.340947 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-metrics-certs\") pod \"84a7170c-6579-469d-a4ed-deb08c89a86f\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " Oct 29 22:56:39.341122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.341045 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a7170c-6579-469d-a4ed-deb08c89a86f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a7170c-6579-469d-a4ed-deb08c89a86f-openshift-service-ca\") pod \"84a7170c-6579-469d-a4ed-deb08c89a86f\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " Oct 29 22:56:39.341122 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.341075 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a7170c-6579-469d-a4ed-deb08c89a86f-kube-api-access-c4tp9\" (UniqueName: \"kubernetes.io/projected/84a7170c-6579-469d-a4ed-deb08c89a86f-kube-api-access-c4tp9\") pod \"84a7170c-6579-469d-a4ed-deb08c89a86f\" (UID: \"84a7170c-6579-469d-a4ed-deb08c89a86f\") " Oct 29 22:56:39.341427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.341381 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a7170c-6579-469d-a4ed-deb08c89a86f-openshift-service-ca" pod "84a7170c-6579-469d-a4ed-deb08c89a86f" (UID: "84a7170c-6579-469d-a4ed-deb08c89a86f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:39.341894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.341869 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a7170c-6579-469d-a4ed-deb08c89a86f-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:39.343184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.343133 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a7170c-6579-469d-a4ed-deb08c89a86f-kube-api-access-c4tp9" pod "84a7170c-6579-469d-a4ed-deb08c89a86f" (UID: "84a7170c-6579-469d-a4ed-deb08c89a86f"). InnerVolumeSpecName "kube-api-access-c4tp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:39.343291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.343200 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-maas-api-tls" pod "84a7170c-6579-469d-a4ed-deb08c89a86f" (UID: "84a7170c-6579-469d-a4ed-deb08c89a86f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:39.343291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.343213 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-metrics-certs" pod "84a7170c-6579-469d-a4ed-deb08c89a86f" (UID: "84a7170c-6579-469d-a4ed-deb08c89a86f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:56:39.360479 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.360453 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:39.443153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.443126 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:39.443153 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.443155 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a7170c-6579-469d-a4ed-deb08c89a86f-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:39.443373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.443170 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4tp9\" (UniqueName: \"kubernetes.io/projected/84a7170c-6579-469d-a4ed-deb08c89a86f-kube-api-access-c4tp9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:39.514008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.513972 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:39.664660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:39.664629 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:40.145290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:40.145256 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" event={"ID":"84a7170c-6579-469d-a4ed-deb08c89a86f","Type":"ContainerDied","Data":"4d5fea59eaf79d19694e21ed33c5511203e9446aea18794e3c59d1e9a96b87e4"} Oct 29 22:56:40.145290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:40.145290 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr" Oct 29 22:56:40.145590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:40.145327 5610 scope.go:122] "RemoveContainer" containerID="498dd05a0729b38bdcc29a0fde9edc6a4284b00f1263dbb574882ed565511d82" Oct 29 22:56:40.192787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:40.192756 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr"] Oct 29 22:56:40.201482 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:40.201453 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-57d336-664c59b858-tfgsr"] Oct 29 22:56:41.028403 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:41.028361 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:41.175756 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:41.175720 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:41.648428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:41.631779 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:41.674612 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:41.674571 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a7170c-6579-469d-a4ed-deb08c89a86f" path="/var/lib/kubelet/pods/84a7170c-6579-469d-a4ed-deb08c89a86f/volumes" Oct 29 22:56:43.370417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:43.357599 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:43.715885 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:56:43.715848 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:56:44.856766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:44.856735 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:45.058286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:45.058254 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:46.225968 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.225937 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:46.415460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.413225 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:46.524999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.524905 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn"] Oct 29 22:56:46.525211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.525180 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" containerID="cri-o://25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966" gracePeriod=30 Oct 29 22:56:46.590907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.590874 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:46.651945 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.651906 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s"] Oct 29 22:56:46.652786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652767 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d1fb92c8-9ef1-4695-bff5-0f3be79033c1" containerName="payload-processing" Oct 29 22:56:46.652786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652787 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fb92c8-9ef1-4695-bff5-0f3be79033c1" containerName="payload-processing" Oct 29 22:56:46.652899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652802 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8191db7b-cb6a-4aae-b00c-c2232aa8f278" containerName="payload-processing" Oct 29 22:56:46.652899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652807 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8191db7b-cb6a-4aae-b00c-c2232aa8f278" containerName="payload-processing" Oct 29 22:56:46.652899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652827 5610 cpu_manager.go:414] "containerMap: removing container" podUID="84a7170c-6579-469d-a4ed-deb08c89a86f" containerName="maas-api" Oct 29 22:56:46.652899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652832 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a7170c-6579-469d-a4ed-deb08c89a86f" containerName="maas-api" Oct 29 22:56:46.652899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652855 5610 cpu_manager.go:414] "containerMap: removing container" podUID="c25ff675-008c-46fa-808f-ea1bf5a30670" containerName="payload-pre-processing" Oct 29 22:56:46.652899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652860 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25ff675-008c-46fa-808f-ea1bf5a30670" containerName="payload-pre-processing" Oct 29 22:56:46.653094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652961 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fb92c8-9ef1-4695-bff5-0f3be79033c1" containerName="payload-processing" Oct 29 22:56:46.653094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652973 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8191db7b-cb6a-4aae-b00c-c2232aa8f278" containerName="payload-processing" Oct 29 22:56:46.653094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652983 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a7170c-6579-469d-a4ed-deb08c89a86f" containerName="maas-api" Oct 29 22:56:46.653094 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.652992 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25ff675-008c-46fa-808f-ea1bf5a30670" containerName="payload-pre-processing" Oct 29 22:56:46.658282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.658264 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.670512 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.670489 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1dabb8-serving-cert\"" Oct 29 22:56:46.679761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.679735 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s"] Oct 29 22:56:46.789104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.789010 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-metrics-certs\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.789277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.789151 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.789277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.789197 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjxl\" (UniqueName: \"kubernetes.io/projected/2bad0630-e43a-48c3-bd90-aecc085ba292-kube-api-access-bqjxl\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.789371 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.789346 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad0630-e43a-48c3-bd90-aecc085ba292-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.862842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.862809 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:46.890602 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.890560 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad0630-e43a-48c3-bd90-aecc085ba292-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.890822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.890802 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-metrics-certs\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.890895 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.890880 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.890963 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.890918 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjxl\" (UniqueName: \"kubernetes.io/projected/2bad0630-e43a-48c3-bd90-aecc085ba292-kube-api-access-bqjxl\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.891334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.891299 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad0630-e43a-48c3-bd90-aecc085ba292-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.893495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.893467 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-metrics-certs\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.893605 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.893502 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.907661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.907638 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjxl\" (UniqueName: \"kubernetes.io/projected/2bad0630-e43a-48c3-bd90-aecc085ba292-kube-api-access-bqjxl\") pod \"maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:46.982026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:46.980768 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:47.018460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:47.016620 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:47.423610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:47.423520 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s"] Oct 29 22:56:47.436476 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:56:47.436445 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bad0630_e43a_48c3_bd90_aecc085ba292.slice/crio-5e4b37a0c2ebaf4c5237154566763c649b4b4c369e6a0df0da6883b0e87025f2 WatchSource:0}: Error finding container 5e4b37a0c2ebaf4c5237154566763c649b4b4c369e6a0df0da6883b0e87025f2: Status 404 returned error can't find the container with id 5e4b37a0c2ebaf4c5237154566763c649b4b4c369e6a0df0da6883b0e87025f2 Oct 29 22:56:47.622418 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:47.618006 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:48.149234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:48.149106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:48.457008 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:48.456975 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" event={"ID":"2bad0630-e43a-48c3-bd90-aecc085ba292","Type":"ContainerStarted","Data":"db2268d564111670a532d3a419fe482a2af97ade2ea3aaa7071ad9220512334c"} Oct 29 22:56:48.457486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:48.457036 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" event={"ID":"2bad0630-e43a-48c3-bd90-aecc085ba292","Type":"ContainerStarted","Data":"5e4b37a0c2ebaf4c5237154566763c649b4b4c369e6a0df0da6883b0e87025f2"} Oct 29 22:56:48.457486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:48.457064 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:48.495713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:48.495629 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" podStartSLOduration=2.156409326 podStartE2EDuration="2.495610468s" totalImagesPullingTime="339.201142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:48.493668784 +0000 UTC m=+2470.974986277" watchObservedRunningTime="2026-10-29 22:56:48.495610468 +0000 UTC m=+2470.976927962" Oct 29 22:56:48.797102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:48.796962 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:50.318916 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:50.318880 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:51.838756 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.838687 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:51.873626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.873597 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:56:51.981880 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.981847 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-envoy\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.982079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.981891 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-credential-socket\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.982079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.981932 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/872887cc-92d3-405f-b405-6d67d4637f18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872887cc-92d3-405f-b405-6d67d4637f18-istio-podinfo\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.982079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.981967 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872887cc-92d3-405f-b405-6d67d4637f18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872887cc-92d3-405f-b405-6d67d4637f18-istiod-ca-cert\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.982471 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.982438 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-credential-socket" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:51.983194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.982662 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-istio-token\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-istio-token\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.983194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.982744 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-kube-api-access-6f56x\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-kube-api-access-6f56x\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.983194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.982839 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-data\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.983194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.982982 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-certs\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.983194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.983042 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-data" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:51.983194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.983132 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-socket\") pod \"872887cc-92d3-405f-b405-6d67d4637f18\" (UID: \"872887cc-92d3-405f-b405-6d67d4637f18\") " Oct 29 22:56:51.985559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.983905 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-certs" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:51.985559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.984029 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872887cc-92d3-405f-b405-6d67d4637f18-istiod-ca-cert" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:56:51.985559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.984230 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-socket" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:51.985559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.984725 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.985559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.984750 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.985559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.984763 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.985559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.984783 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:51.985559 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:51.984797 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872887cc-92d3-405f-b405-6d67d4637f18-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.008918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.008876 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-envoy" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:56:52.010547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.010515 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/872887cc-92d3-405f-b405-6d67d4637f18-istio-podinfo" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:56:52.011741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.011562 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-istio-token" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:52.012705 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.012680 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-kube-api-access-6f56x" pod "872887cc-92d3-405f-b405-6d67d4637f18" (UID: "872887cc-92d3-405f-b405-6d67d4637f18"). InnerVolumeSpecName "kube-api-access-6f56x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:56:52.086316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.086237 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872887cc-92d3-405f-b405-6d67d4637f18-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.086316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.086268 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872887cc-92d3-405f-b405-6d67d4637f18-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.086316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.086277 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.086316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.086286 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f56x\" (UniqueName: \"kubernetes.io/projected/872887cc-92d3-405f-b405-6d67d4637f18-kube-api-access-6f56x\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:56:52.494179 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.494144 5610 generic.go:356] "Generic (PLEG): container finished" podID="872887cc-92d3-405f-b405-6d67d4637f18" containerID="25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966" exitCode=0 Oct 29 22:56:52.494360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.494188 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" event={"ID":"872887cc-92d3-405f-b405-6d67d4637f18","Type":"ContainerDied","Data":"25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966"} Oct 29 22:56:52.494360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.494221 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" Oct 29 22:56:52.494360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.494296 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn" event={"ID":"872887cc-92d3-405f-b405-6d67d4637f18","Type":"ContainerDied","Data":"d6d571236df8882fa285d761350ef993a45f3adfd3eb89b144ea4f6aa8d37b2b"} Oct 29 22:56:52.494360 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.494328 5610 scope.go:122] "RemoveContainer" containerID="25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966" Oct 29 22:56:52.505733 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.505462 5610 scope.go:122] "RemoveContainer" containerID="25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966" Oct 29 22:56:52.506121 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:56:52.506088 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966\": container with ID starting with 25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966 not found: ID does not exist" containerID="25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966" Oct 29 22:56:52.506372 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.506342 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966"} err="failed to get container status \"25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966\": rpc error: code = NotFound desc = could not find container \"25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966\": container with ID starting with 25715f87e33219b4934b2b9ca50b32c479d319fb99491aeee1dfd3fb8e9ed966 not found: ID does not exist" Oct 29 22:56:52.551281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.551252 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn"] Oct 29 22:56:52.561055 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:52.561028 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-57d336-openshift-default-854545cc57-jgfjn"] Oct 29 22:56:53.637369 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:53.637338 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872887cc-92d3-405f-b405-6d67d4637f18" path="/var/lib/kubelet/pods/872887cc-92d3-405f-b405-6d67d4637f18/volumes" Oct 29 22:56:53.961524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:53.961477 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:54.471184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:54.471149 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 22:56:55.429415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:55.423676 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:56.910933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:56.910896 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:56:58.444671 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:56:58.444572 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:00.311902 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:00.311870 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:01.598804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:01.594601 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:01.775838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:01.775783 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:02.156427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.154791 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf"] Oct 29 22:57:02.156427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.156224 5610 cpu_manager.go:414] "containerMap: removing container" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" Oct 29 22:57:02.156427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.156247 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" Oct 29 22:57:02.156775 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.156500 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="872887cc-92d3-405f-b405-6d67d4637f18" containerName="istio-proxy" Oct 29 22:57:02.226885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.226852 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf"] Oct 29 22:57:02.227187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.227172 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:02.243450 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.243418 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09fc89\"" Oct 29 22:57:02.253813 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.252427 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09fc89-dockercfg-92qp7\"" Oct 29 22:57:02.306044 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.306007 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch"] Oct 29 22:57:02.312093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.312071 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:02.320518 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.320492 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch"] Oct 29 22:57:02.326788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.326742 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld877\" (UniqueName: \"kubernetes.io/projected/9c999451-dd15-4eac-8946-c5e0e7c060a5-kube-api-access-ld877\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch\" (UID: \"9c999451-dd15-4eac-8946-c5e0e7c060a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:02.326996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.326972 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4ht\" (UniqueName: \"kubernetes.io/projected/22dbafd3-5640-415b-b8ab-248aa8079d63-kube-api-access-jf4ht\") pod \"payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf\" (UID: \"22dbafd3-5640-415b-b8ab-248aa8079d63\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:02.327140 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.327121 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dbafd3-5640-415b-b8ab-248aa8079d63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf\" (UID: \"22dbafd3-5640-415b-b8ab-248aa8079d63\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:02.327333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.327313 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c999451-dd15-4eac-8946-c5e0e7c060a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch\" (UID: \"9c999451-dd15-4eac-8946-c5e0e7c060a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:02.428438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.428333 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dbafd3-5640-415b-b8ab-248aa8079d63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf\" (UID: \"22dbafd3-5640-415b-b8ab-248aa8079d63\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:02.428613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.428487 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c999451-dd15-4eac-8946-c5e0e7c060a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch\" (UID: \"9c999451-dd15-4eac-8946-c5e0e7c060a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:02.428717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.428701 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld877\" (UniqueName: \"kubernetes.io/projected/9c999451-dd15-4eac-8946-c5e0e7c060a5-kube-api-access-ld877\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch\" (UID: \"9c999451-dd15-4eac-8946-c5e0e7c060a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:02.428769 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.428749 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4ht\" (UniqueName: \"kubernetes.io/projected/22dbafd3-5640-415b-b8ab-248aa8079d63-kube-api-access-jf4ht\") pod \"payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf\" (UID: \"22dbafd3-5640-415b-b8ab-248aa8079d63\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:02.429437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.429379 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c999451-dd15-4eac-8946-c5e0e7c060a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch\" (UID: \"9c999451-dd15-4eac-8946-c5e0e7c060a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:02.429514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.429379 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dbafd3-5640-415b-b8ab-248aa8079d63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf\" (UID: \"22dbafd3-5640-415b-b8ab-248aa8079d63\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:02.445985 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.445955 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld877\" (UniqueName: \"kubernetes.io/projected/9c999451-dd15-4eac-8946-c5e0e7c060a5-kube-api-access-ld877\") pod \"payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch\" (UID: \"9c999451-dd15-4eac-8946-c5e0e7c060a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:02.446141 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.446089 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4ht\" (UniqueName: \"kubernetes.io/projected/22dbafd3-5640-415b-b8ab-248aa8079d63-kube-api-access-jf4ht\") pod \"payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf\" (UID: \"22dbafd3-5640-415b-b8ab-248aa8079d63\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:02.603603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.603569 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:02.632083 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.632057 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:02.930575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.930544 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:02.941653 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:02.940976 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf"] Oct 29 22:57:02.947371 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:57:02.947337 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dbafd3_5640_415b_b8ab_248aa8079d63.slice/crio-c2259580918feac044011beb5c6efee1b15bcb6e6ccb0896bc526fe8d909e646 WatchSource:0}: Error finding container c2259580918feac044011beb5c6efee1b15bcb6e6ccb0896bc526fe8d909e646: Status 404 returned error can't find the container with id c2259580918feac044011beb5c6efee1b15bcb6e6ccb0896bc526fe8d909e646 Oct 29 22:57:03.135541 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.135505 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch"] Oct 29 22:57:03.137528 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:57:03.137202 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c999451_dd15_4eac_8946_c5e0e7c060a5.slice/crio-91e3bacdd021ca151627818de2e6fe3c77cc305c1636b7e57a82ad3500f37ba6 WatchSource:0}: Error finding container 91e3bacdd021ca151627818de2e6fe3c77cc305c1636b7e57a82ad3500f37ba6: Status 404 returned error can't find the container with id 91e3bacdd021ca151627818de2e6fe3c77cc305c1636b7e57a82ad3500f37ba6 Oct 29 22:57:03.638026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.637943 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" event={"ID":"22dbafd3-5640-415b-b8ab-248aa8079d63","Type":"ContainerStarted","Data":"f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1"} Oct 29 22:57:03.638026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.637998 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" event={"ID":"22dbafd3-5640-415b-b8ab-248aa8079d63","Type":"ContainerStarted","Data":"c2259580918feac044011beb5c6efee1b15bcb6e6ccb0896bc526fe8d909e646"} Oct 29 22:57:03.638026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.638014 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" event={"ID":"9c999451-dd15-4eac-8946-c5e0e7c060a5","Type":"ContainerStarted","Data":"8398f0b32ebd3778d6239d2f710cf07ee35c8712a176c3a20a05ec886b89d874"} Oct 29 22:57:03.638026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.638029 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" event={"ID":"9c999451-dd15-4eac-8946-c5e0e7c060a5","Type":"ContainerStarted","Data":"91e3bacdd021ca151627818de2e6fe3c77cc305c1636b7e57a82ad3500f37ba6"} Oct 29 22:57:03.638520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.638047 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:03.638520 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.638096 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:03.684787 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.684726 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" podStartSLOduration=1.684709456 podStartE2EDuration="1.684709456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:03.663895649 +0000 UTC m=+2486.145213143" watchObservedRunningTime="2026-10-29 22:57:03.684709456 +0000 UTC m=+2486.166026950" Oct 29 22:57:03.702934 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:03.702876 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" podStartSLOduration=1.702846177 podStartE2EDuration="1.702846177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:03.702059581 +0000 UTC m=+2486.183377079" watchObservedRunningTime="2026-10-29 22:57:03.702846177 +0000 UTC m=+2486.184163671" Oct 29 22:57:04.669842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:04.669783 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:06.324529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:06.324494 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:07.931537 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:07.926374 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:09.515338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:09.515298 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:11.227496 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:11.227382 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:11.710290 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:11.710254 5610 generic.go:356] "Generic (PLEG): container finished" podID="89be13e4-0548-49e6-8c81-006939d66752" containerID="28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9" exitCode=0 Oct 29 22:57:11.710488 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:11.710325 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" event={"ID":"89be13e4-0548-49e6-8c81-006939d66752","Type":"ContainerDied","Data":"28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9"} Oct 29 22:57:11.710776 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:11.710761 5610 scope.go:122] "RemoveContainer" containerID="28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9" Oct 29 22:57:11.712646 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:11.712621 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:11.712736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:11.712692 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:11.712736 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:11.712724 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:57:12.720820 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:12.720783 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" event={"ID":"89be13e4-0548-49e6-8c81-006939d66752","Type":"ContainerStarted","Data":"56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21"} Oct 29 22:57:13.063995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:13.063901 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:13.348675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:13.348590 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:57:13.349711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:13.349685 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:57:13.349842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:13.349730 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 29 22:57:13.580535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:13.580500 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:57:13.724579 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:57:13.724542 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:57:14.348090 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:14.348053 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:57:14.348276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:14.348130 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 29 22:57:14.645461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:14.645356 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:57:14.645461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:14.645441 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:57:14.680222 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:14.680167 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:15.348353 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:15.348323 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:57:15.348774 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:15.348370 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 29 22:57:16.329466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:16.329427 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:16.352438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:16.348557 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:57:16.352438 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:16.348647 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 29 22:57:17.317583 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:17.317548 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:17.348255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:17.348224 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 29 22:57:17.348442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:17.348271 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 29 22:57:18.282550 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:18.282502 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:18.353528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:18.353500 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:57:18.354771 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:18.354751 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 22:57:19.792735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:19.792672 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:21.233486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:21.233442 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:21.726411 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:21.726353 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:23.447725 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:23.445651 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:24.856704 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:24.856620 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:25.271017 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:25.270970 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:25.447727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:25.447694 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t"] Oct 29 22:57:25.673686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:25.673604 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:25.833461 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:25.833418 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:26.303732 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:26.303685 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:26.802377 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:26.802348 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:26.960841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:26.960807 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:28.666249 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:28.666190 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:30.262307 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.262272 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:30.451506 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.451464 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:30.451868 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.451497 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" podUID="9f5e0990-dc13-4f97-97db-f21eec274ff6" containerName="payload-processing" containerID="cri-o://3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc" gracePeriod=25 Oct 29 22:57:30.785811 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.785781 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:57:30.909180 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.909089 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5e0990-dc13-4f97-97db-f21eec274ff6-kube-api-access-jzdzd\" (UniqueName: \"kubernetes.io/projected/9f5e0990-dc13-4f97-97db-f21eec274ff6-kube-api-access-jzdzd\") pod \"9f5e0990-dc13-4f97-97db-f21eec274ff6\" (UID: \"9f5e0990-dc13-4f97-97db-f21eec274ff6\") " Oct 29 22:57:30.909362 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.909194 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5e0990-dc13-4f97-97db-f21eec274ff6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5e0990-dc13-4f97-97db-f21eec274ff6-plugins-config-volume\") pod \"9f5e0990-dc13-4f97-97db-f21eec274ff6\" (UID: \"9f5e0990-dc13-4f97-97db-f21eec274ff6\") " Oct 29 22:57:30.909657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.909631 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5e0990-dc13-4f97-97db-f21eec274ff6-plugins-config-volume" pod "9f5e0990-dc13-4f97-97db-f21eec274ff6" (UID: "9f5e0990-dc13-4f97-97db-f21eec274ff6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:30.909889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.909872 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f5e0990-dc13-4f97-97db-f21eec274ff6-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:30.912057 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.912025 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5e0990-dc13-4f97-97db-f21eec274ff6-kube-api-access-jzdzd" pod "9f5e0990-dc13-4f97-97db-f21eec274ff6" (UID: "9f5e0990-dc13-4f97-97db-f21eec274ff6"). InnerVolumeSpecName "kube-api-access-jzdzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:30.973609 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.973570 5610 generic.go:356] "Generic (PLEG): container finished" podID="9f5e0990-dc13-4f97-97db-f21eec274ff6" containerID="3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc" exitCode=0 Oct 29 22:57:30.973765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.973623 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" Oct 29 22:57:30.973765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.973654 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" event={"ID":"9f5e0990-dc13-4f97-97db-f21eec274ff6","Type":"ContainerDied","Data":"3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc"} Oct 29 22:57:30.973765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.973715 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t" event={"ID":"9f5e0990-dc13-4f97-97db-f21eec274ff6","Type":"ContainerDied","Data":"b8313b40b02b171274e1436bc20128858f889264c33122d36b6cd87f234779e9"} Oct 29 22:57:30.973765 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.973744 5610 scope.go:122] "RemoveContainer" containerID="3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc" Oct 29 22:57:30.985109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.985088 5610 scope.go:122] "RemoveContainer" containerID="3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc" Oct 29 22:57:30.985450 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:57:30.985431 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc\": container with ID starting with 3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc not found: ID does not exist" containerID="3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc" Oct 29 22:57:30.985526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:30.985458 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc"} err="failed to get container status \"3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc\": rpc error: code = NotFound desc = could not find container \"3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc\": container with ID starting with 3d328e6db76b436d52acb56da217d8c5bc69dca3dafe12d1d150a552f1ab22bc not found: ID does not exist" Oct 29 22:57:31.011796 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:31.011766 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdzd\" (UniqueName: \"kubernetes.io/projected/9f5e0990-dc13-4f97-97db-f21eec274ff6-kube-api-access-jzdzd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:31.029033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:31.029007 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t"] Oct 29 22:57:31.035926 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:31.035904 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-995d7bdc4-hvw9t"] Oct 29 22:57:31.642192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:31.642152 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5e0990-dc13-4f97-97db-f21eec274ff6" path="/var/lib/kubelet/pods/9f5e0990-dc13-4f97-97db-f21eec274ff6/volumes" Oct 29 22:57:32.024643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.024601 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:32.123514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.123482 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m"] Oct 29 22:57:32.124328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.124308 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9f5e0990-dc13-4f97-97db-f21eec274ff6" containerName="payload-processing" Oct 29 22:57:32.124423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.124328 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5e0990-dc13-4f97-97db-f21eec274ff6" containerName="payload-processing" Oct 29 22:57:32.124503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.124492 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5e0990-dc13-4f97-97db-f21eec274ff6" containerName="payload-processing" Oct 29 22:57:32.129102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.129083 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:32.149012 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.148987 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m"] Oct 29 22:57:32.155069 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.155044 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bcb810-dockercfg-ppjtv\"" Oct 29 22:57:32.155204 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.155157 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bcb810\"" Oct 29 22:57:32.216428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.216378 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs"] Oct 29 22:57:32.221089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.221068 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:32.228473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.228443 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4whm\" (UniqueName: \"kubernetes.io/projected/189a79c6-509e-4db6-bf58-0639630a4e9b-kube-api-access-v4whm\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m\" (UID: \"189a79c6-509e-4db6-bf58-0639630a4e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:32.228766 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.228744 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189a79c6-509e-4db6-bf58-0639630a4e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m\" (UID: \"189a79c6-509e-4db6-bf58-0639630a4e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:32.268998 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.268964 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs"] Oct 29 22:57:32.279285 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.279179 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:32.330336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.330303 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4whm\" (UniqueName: \"kubernetes.io/projected/189a79c6-509e-4db6-bf58-0639630a4e9b-kube-api-access-v4whm\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m\" (UID: \"189a79c6-509e-4db6-bf58-0639630a4e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:32.330597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.330466 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pscbz\" (UniqueName: \"kubernetes.io/projected/3229ddc3-6a63-4629-9ef0-1b57478ddf45-kube-api-access-pscbz\") pod \"payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs\" (UID: \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:32.330597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.330565 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3229ddc3-6a63-4629-9ef0-1b57478ddf45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs\" (UID: \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:32.330744 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.330599 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189a79c6-509e-4db6-bf58-0639630a4e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m\" (UID: \"189a79c6-509e-4db6-bf58-0639630a4e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:32.331346 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.331322 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189a79c6-509e-4db6-bf58-0639630a4e9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m\" (UID: \"189a79c6-509e-4db6-bf58-0639630a4e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:32.357698 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.357671 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4whm\" (UniqueName: \"kubernetes.io/projected/189a79c6-509e-4db6-bf58-0639630a4e9b-kube-api-access-v4whm\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m\" (UID: \"189a79c6-509e-4db6-bf58-0639630a4e9b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:32.431758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.431725 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pscbz\" (UniqueName: \"kubernetes.io/projected/3229ddc3-6a63-4629-9ef0-1b57478ddf45-kube-api-access-pscbz\") pod \"payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs\" (UID: \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:32.431933 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.431787 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3229ddc3-6a63-4629-9ef0-1b57478ddf45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs\" (UID: \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:32.432504 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.432478 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3229ddc3-6a63-4629-9ef0-1b57478ddf45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs\" (UID: \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:32.443849 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.443824 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:32.450229 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.450204 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pscbz\" (UniqueName: \"kubernetes.io/projected/3229ddc3-6a63-4629-9ef0-1b57478ddf45-kube-api-access-pscbz\") pod \"payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs\" (UID: \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:32.467603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.467538 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:32.586842 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.565677 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:32.976996 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:32.976968 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m"] Oct 29 22:57:32.978987 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:57:32.978959 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189a79c6_509e_4db6_bf58_0639630a4e9b.slice/crio-40f1be4d8e6d0fe7d4ff3d5501dd658080730b0d38534088e7e5253489803178 WatchSource:0}: Error finding container 40f1be4d8e6d0fe7d4ff3d5501dd658080730b0d38534088e7e5253489803178: Status 404 returned error can't find the container with id 40f1be4d8e6d0fe7d4ff3d5501dd658080730b0d38534088e7e5253489803178 Oct 29 22:57:33.021587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:33.021550 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" event={"ID":"189a79c6-509e-4db6-bf58-0639630a4e9b","Type":"ContainerStarted","Data":"40f1be4d8e6d0fe7d4ff3d5501dd658080730b0d38534088e7e5253489803178"} Oct 29 22:57:33.085515 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:57:33.085476 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3229ddc3_6a63_4629_9ef0_1b57478ddf45.slice/crio-00dc41fa18939c03781fa95324e5190ad80a995eea124c6744db57fa8a4d6422 WatchSource:0}: Error finding container 00dc41fa18939c03781fa95324e5190ad80a995eea124c6744db57fa8a4d6422: Status 404 returned error can't find the container with id 00dc41fa18939c03781fa95324e5190ad80a995eea124c6744db57fa8a4d6422 Oct 29 22:57:33.086694 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:33.086663 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs"] Oct 29 22:57:34.039339 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:34.039301 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" event={"ID":"3229ddc3-6a63-4629-9ef0-1b57478ddf45","Type":"ContainerStarted","Data":"504c5325e159b5e9fa099ad86545cd5ab3d6b200d04f33e06ff0afa7e35eb06f"} Oct 29 22:57:34.039843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:34.039362 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" event={"ID":"3229ddc3-6a63-4629-9ef0-1b57478ddf45","Type":"ContainerStarted","Data":"00dc41fa18939c03781fa95324e5190ad80a995eea124c6744db57fa8a4d6422"} Oct 29 22:57:34.039843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:34.039712 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:34.042455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:34.041852 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" event={"ID":"189a79c6-509e-4db6-bf58-0639630a4e9b","Type":"ContainerStarted","Data":"3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847"} Oct 29 22:57:34.042455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:34.042384 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:34.081026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:34.080965 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" podStartSLOduration=2.080949484 podStartE2EDuration="2.080949484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:34.07902188 +0000 UTC m=+2516.560339371" watchObservedRunningTime="2026-10-29 22:57:34.080949484 +0000 UTC m=+2516.562266976" Oct 29 22:57:34.118489 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:34.118443 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" podStartSLOduration=2.118429752 podStartE2EDuration="2.118429752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:34.114290538 +0000 UTC m=+2516.595608041" watchObservedRunningTime="2026-10-29 22:57:34.118429752 +0000 UTC m=+2516.599747245" Oct 29 22:57:34.345590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:34.339354 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:35.943422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:35.931813 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:36.821538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:36.821503 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:36.982078 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:36.982044 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:37.175115 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.175036 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:37.229898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.229865 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb"] Oct 29 22:57:37.230252 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.230102 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" podUID="daf75a04-e99c-45c3-a485-611120dda4b0" containerName="maas-api" containerID="cri-o://350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb" gracePeriod=30 Oct 29 22:57:37.363907 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.363871 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:37.402244 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.402207 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw"] Oct 29 22:57:37.402540 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.402490 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" podUID="31921b2e-8bdb-47be-ac76-fa83e4607ff4" containerName="payload-processing" containerID="cri-o://bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e" gracePeriod=30 Oct 29 22:57:37.559033 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.558840 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:37.564590 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.564212 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:57:37.609937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.608894 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-metrics-certs\") pod \"daf75a04-e99c-45c3-a485-611120dda4b0\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " Oct 29 22:57:37.609937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.609159 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf75a04-e99c-45c3-a485-611120dda4b0-kube-api-access-7c6np\" (UniqueName: \"kubernetes.io/projected/daf75a04-e99c-45c3-a485-611120dda4b0-kube-api-access-7c6np\") pod \"daf75a04-e99c-45c3-a485-611120dda4b0\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " Oct 29 22:57:37.609937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.609259 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls\") pod \"daf75a04-e99c-45c3-a485-611120dda4b0\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " Oct 29 22:57:37.609937 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.609352 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf75a04-e99c-45c3-a485-611120dda4b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf75a04-e99c-45c3-a485-611120dda4b0-openshift-service-ca\") pod \"daf75a04-e99c-45c3-a485-611120dda4b0\" (UID: \"daf75a04-e99c-45c3-a485-611120dda4b0\") " Oct 29 22:57:37.610604 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.610485 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf75a04-e99c-45c3-a485-611120dda4b0-openshift-service-ca" pod "daf75a04-e99c-45c3-a485-611120dda4b0" (UID: "daf75a04-e99c-45c3-a485-611120dda4b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:37.620841 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.620804 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-metrics-certs" pod "daf75a04-e99c-45c3-a485-611120dda4b0" (UID: "daf75a04-e99c-45c3-a485-611120dda4b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:37.642099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.642039 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf75a04-e99c-45c3-a485-611120dda4b0-kube-api-access-7c6np" pod "daf75a04-e99c-45c3-a485-611120dda4b0" (UID: "daf75a04-e99c-45c3-a485-611120dda4b0"). InnerVolumeSpecName "kube-api-access-7c6np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:37.642965 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.642937 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls" pod "daf75a04-e99c-45c3-a485-611120dda4b0" (UID: "daf75a04-e99c-45c3-a485-611120dda4b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 22:57:37.712553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.712284 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6np\" (UniqueName: \"kubernetes.io/projected/daf75a04-e99c-45c3-a485-611120dda4b0-kube-api-access-7c6np\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:37.712553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.712316 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:37.712553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.712331 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf75a04-e99c-45c3-a485-611120dda4b0-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:37.712553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.712345 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf75a04-e99c-45c3-a485-611120dda4b0-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:37.807423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.749136 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:37.839503 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.839447 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:57:37.930415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.924709 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31921b2e-8bdb-47be-ac76-fa83e4607ff4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921b2e-8bdb-47be-ac76-fa83e4607ff4-plugins-config-volume\") pod \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\" (UID: \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\") " Oct 29 22:57:37.930415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.925009 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31921b2e-8bdb-47be-ac76-fa83e4607ff4-kube-api-access-n7sch\" (UniqueName: \"kubernetes.io/projected/31921b2e-8bdb-47be-ac76-fa83e4607ff4-kube-api-access-n7sch\") pod \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\" (UID: \"31921b2e-8bdb-47be-ac76-fa83e4607ff4\") " Oct 29 22:57:37.931193 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.931148 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31921b2e-8bdb-47be-ac76-fa83e4607ff4-plugins-config-volume" pod "31921b2e-8bdb-47be-ac76-fa83e4607ff4" (UID: "31921b2e-8bdb-47be-ac76-fa83e4607ff4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:37.947487 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.946770 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31921b2e-8bdb-47be-ac76-fa83e4607ff4-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:37.986463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:37.974745 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:38.006636 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.006567 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31921b2e-8bdb-47be-ac76-fa83e4607ff4-kube-api-access-n7sch" pod "31921b2e-8bdb-47be-ac76-fa83e4607ff4" (UID: "31921b2e-8bdb-47be-ac76-fa83e4607ff4"). InnerVolumeSpecName "kube-api-access-n7sch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:38.048070 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.048039 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7sch\" (UniqueName: \"kubernetes.io/projected/31921b2e-8bdb-47be-ac76-fa83e4607ff4-kube-api-access-n7sch\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:38.077145 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.077105 5610 generic.go:356] "Generic (PLEG): container finished" podID="31921b2e-8bdb-47be-ac76-fa83e4607ff4" containerID="bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e" exitCode=0 Oct 29 22:57:38.077366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.077169 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" Oct 29 22:57:38.077366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.077266 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" event={"ID":"31921b2e-8bdb-47be-ac76-fa83e4607ff4","Type":"ContainerDied","Data":"bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e"} Oct 29 22:57:38.077366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.077344 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw" event={"ID":"31921b2e-8bdb-47be-ac76-fa83e4607ff4","Type":"ContainerDied","Data":"4db93bb10514335f3adc2d37b0c03b25b4691003710550a2fd06e86e3ac82861"} Oct 29 22:57:38.077619 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.077373 5610 scope.go:122] "RemoveContainer" containerID="bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e" Oct 29 22:57:38.081960 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.081930 5610 generic.go:356] "Generic (PLEG): container finished" podID="daf75a04-e99c-45c3-a485-611120dda4b0" containerID="350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb" exitCode=0 Oct 29 22:57:38.082118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.082021 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" event={"ID":"daf75a04-e99c-45c3-a485-611120dda4b0","Type":"ContainerDied","Data":"350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb"} Oct 29 22:57:38.082118 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.082063 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" event={"ID":"daf75a04-e99c-45c3-a485-611120dda4b0","Type":"ContainerDied","Data":"7490783d1da3722b66c9317af6373171804da404c6efaafd377cf5982295fe98"} Oct 29 22:57:38.082241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.082156 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb" Oct 29 22:57:38.092072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.092049 5610 scope.go:122] "RemoveContainer" containerID="bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e" Oct 29 22:57:38.092350 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:57:38.092324 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e\": container with ID starting with bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e not found: ID does not exist" containerID="bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e" Oct 29 22:57:38.092460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.092362 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e"} err="failed to get container status \"bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e\": rpc error: code = NotFound desc = could not find container \"bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e\": container with ID starting with bca2cc86bf04ee1e32c28feb3298abee0b2e49a9f5ff2c1ee37c231cef4b134e not found: ID does not exist" Oct 29 22:57:38.092460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.092385 5610 scope.go:122] "RemoveContainer" containerID="350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb" Oct 29 22:57:38.106840 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:57:38.106814 5610 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31921b2e_8bdb_47be_ac76_fa83e4607ff4.slice\": RecentStats: unable to find data in memory cache]" Oct 29 22:57:38.109631 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.109609 5610 scope.go:122] "RemoveContainer" containerID="350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb" Oct 29 22:57:38.110111 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:57:38.110082 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb\": container with ID starting with 350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb not found: ID does not exist" containerID="350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb" Oct 29 22:57:38.110238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.110122 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb"} err="failed to get container status \"350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb\": rpc error: code = NotFound desc = could not find container \"350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb\": container with ID starting with 350083ad610adf5ecd272cc08ed55a49afd4a4d645de87ca0f3da82760427efb not found: ID does not exist" Oct 29 22:57:38.136370 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.136341 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb"] Oct 29 22:57:38.141089 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.141065 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a5fe-77f69d5d85-jj9gb"] Oct 29 22:57:38.176848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.176815 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:38.180727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.180702 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw"] Oct 29 22:57:38.188894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.188868 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a5fe-84b8dd9794-jsxnw"] Oct 29 22:57:38.377173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.377135 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:38.775675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:38.775641 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:39.638027 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:39.637982 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31921b2e-8bdb-47be-ac76-fa83e4607ff4" path="/var/lib/kubelet/pods/31921b2e-8bdb-47be-ac76-fa83e4607ff4/volumes" Oct 29 22:57:39.638717 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:39.638694 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf75a04-e99c-45c3-a485-611120dda4b0" path="/var/lib/kubelet/pods/daf75a04-e99c-45c3-a485-611120dda4b0/volumes" Oct 29 22:57:40.183837 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:40.183805 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:40.437538 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:40.437462 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:41.821441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:41.818108 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:43.565009 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:43.564972 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:43.715117 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:57:43.715080 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:57:43.803071 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:43.803038 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:45.052183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:45.052153 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:57:45.506459 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:45.506425 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:46.061465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:46.061435 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:57:46.191035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:46.190987 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:46.383927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:46.383810 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:46.483643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:46.483602 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg"] Oct 29 22:57:46.483940 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:46.483909 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" containerID="cri-o://c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e" gracePeriod=30 Oct 29 22:57:46.605416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:46.592725 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:48.156702 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:48.156668 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:49.250959 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:49.250921 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 22:57:49.251373 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:49.250980 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 22:57:50.234475 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:50.232602 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:51.772135 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.772075 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:57:51.895896 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.895856 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-podinfo\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.895914 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-token\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-token\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.895987 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-envoy\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896098 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896062 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-socket\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896127 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-kube-api-access-btcc4\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-kube-api-access-btcc4\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896276 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896254 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-data\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896297 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-certs\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896365 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-socket" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:57:51.896387 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896407 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-credential-socket\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896438 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istiod-ca-cert\") pod \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\" (UID: \"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800\") " Oct 29 22:57:51.896781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896688 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-certs" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:57:51.896781 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896706 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-credential-socket" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:57:51.896953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896879 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-data" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:57:51.897016 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.896990 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istiod-ca-cert" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:57:51.897591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.897357 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.897591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.897379 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.897591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.897432 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.897591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.897449 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.897591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.897463 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.898804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.898776 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-envoy" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 22:57:51.898924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.898818 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-token" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:51.898924 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.898844 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-kube-api-access-btcc4" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "kube-api-access-btcc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:57:51.899060 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.899042 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-podinfo" pod "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" (UID: "b7eef245-d0ec-4b6c-a7e1-2a2c4e806800"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 22:57:51.998616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.998583 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.998616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.998613 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.998616 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.998624 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:51.998897 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:51.998633 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcc4\" (UniqueName: \"kubernetes.io/projected/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800-kube-api-access-btcc4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:57:52.295627 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.295594 5610 generic.go:356] "Generic (PLEG): container finished" podID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerID="c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e" exitCode=0 Oct 29 22:57:52.295838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.295664 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" Oct 29 22:57:52.295838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.295657 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" event={"ID":"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800","Type":"ContainerDied","Data":"c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e"} Oct 29 22:57:52.295838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.295767 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg" event={"ID":"b7eef245-d0ec-4b6c-a7e1-2a2c4e806800","Type":"ContainerDied","Data":"a62c28c6971198a757746110a977e6038833e3d25cacf84a63ef458509915ee5"} Oct 29 22:57:52.295838 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.295792 5610 scope.go:122] "RemoveContainer" containerID="c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e" Oct 29 22:57:52.302273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.302246 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:52.357898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.357868 5610 scope.go:122] "RemoveContainer" containerID="41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c" Oct 29 22:57:52.380152 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.380128 5610 scope.go:122] "RemoveContainer" containerID="c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e" Oct 29 22:57:52.380569 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:57:52.380529 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e\": container with ID starting with c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e not found: ID does not exist" containerID="c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e" Oct 29 22:57:52.380677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.380581 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e"} err="failed to get container status \"c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e\": rpc error: code = NotFound desc = could not find container \"c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e\": container with ID starting with c47c9983002f5ed53c1b97668430ec6c9947bfbf82327d0c8a2554b57497869e not found: ID does not exist" Oct 29 22:57:52.380677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.380607 5610 scope.go:122] "RemoveContainer" containerID="41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c" Oct 29 22:57:52.380899 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:57:52.380877 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c\": container with ID starting with 41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c not found: ID does not exist" containerID="41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c" Oct 29 22:57:52.380972 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.380908 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c"} err="failed to get container status \"41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c\": rpc error: code = NotFound desc = could not find container \"41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c\": container with ID starting with 41aff5a9f2447c73aa28b656054821c5bf8ac54dd276e5b7a897d975df4ed42c not found: ID does not exist" Oct 29 22:57:52.389263 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.389239 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg"] Oct 29 22:57:52.392948 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:52.392916 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a5fe-openshift-default-b997867f9-rbpbg"] Oct 29 22:57:53.638535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:53.638498 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" path="/var/lib/kubelet/pods/b7eef245-d0ec-4b6c-a7e1-2a2c4e806800/volumes" Oct 29 22:57:53.969626 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:53.955996 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:55.689790 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:55.688836 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:57.258827 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:57.258756 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:57:59.076522 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:57:59.076481 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:00.613234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:00.613193 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:02.173437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:02.173375 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:04.179291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:04.179252 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:05.137972 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.137937 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9"] Oct 29 22:58:05.138727 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138712 5610 cpu_manager.go:414] "containerMap: removing container" podUID="daf75a04-e99c-45c3-a485-611120dda4b0" containerName="maas-api" Oct 29 22:58:05.138778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138728 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf75a04-e99c-45c3-a485-611120dda4b0" containerName="maas-api" Oct 29 22:58:05.138778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138747 5610 cpu_manager.go:414] "containerMap: removing container" podUID="31921b2e-8bdb-47be-ac76-fa83e4607ff4" containerName="payload-processing" Oct 29 22:58:05.138778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138752 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31921b2e-8bdb-47be-ac76-fa83e4607ff4" containerName="payload-processing" Oct 29 22:58:05.138885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138786 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" Oct 29 22:58:05.138885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138792 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" Oct 29 22:58:05.138885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138814 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" Oct 29 22:58:05.138885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138819 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" Oct 29 22:58:05.139020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138905 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" Oct 29 22:58:05.139020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138917 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf75a04-e99c-45c3-a485-611120dda4b0" containerName="maas-api" Oct 29 22:58:05.139020 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.138926 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="31921b2e-8bdb-47be-ac76-fa83e4607ff4" containerName="payload-processing" Oct 29 22:58:05.144992 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.144970 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.167643 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.167606 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9"] Oct 29 22:58:05.173420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.170657 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8f1627\"/\"auth-test-model-8f1627-kserve-self-signed-certs\"" Oct 29 22:58:05.173420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.170778 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8f1627\"/\"default-dockercfg-2nnkm\"" Oct 29 22:58:05.173420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.171531 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8f1627\"/\"kube-root-ca.crt\"" Oct 29 22:58:05.173420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.171579 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8f1627\"/\"openshift-service-ca.crt\"" Oct 29 22:58:05.205119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.205088 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-home\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.205628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.205146 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-dshm\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.205628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.205187 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wf4t\" (UniqueName: \"kubernetes.io/projected/f97607e1-3c37-4b30-99ef-e9d2fe142938-kube-api-access-9wf4t\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.205628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.205219 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-tmp-dir\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.205628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.205315 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97607e1-3c37-4b30-99ef-e9d2fe142938-tls-certs\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.205628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.205379 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-model-cache\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.270950 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.270918 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:05.307273 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.307239 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-model-cache\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.307466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.307341 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-home\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.307466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.307409 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-dshm\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.307466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.307461 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wf4t\" (UniqueName: \"kubernetes.io/projected/f97607e1-3c37-4b30-99ef-e9d2fe142938-kube-api-access-9wf4t\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.307660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.307504 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-tmp-dir\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.307660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.307647 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97607e1-3c37-4b30-99ef-e9d2fe142938-tls-certs\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.307770 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.307707 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-model-cache\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.308002 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.307959 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-tmp-dir\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.308116 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.308075 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-home\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.310325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.310299 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97607e1-3c37-4b30-99ef-e9d2fe142938-tls-certs\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.310463 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.310441 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-dshm\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.326553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.326532 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wf4t\" (UniqueName: \"kubernetes.io/projected/f97607e1-3c37-4b30-99ef-e9d2fe142938-kube-api-access-9wf4t\") pod \"auth-test-model-8f1627-kserve-6ddb4675b5-smhp9\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.474087 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.474053 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:05.831599 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.831573 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9"] Oct 29 22:58:05.832924 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:58:05.832902 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf97607e1_3c37_4b30_99ef_e9d2fe142938.slice/crio-5e888d3acbf9145c1b2d22546787ead5f79d27bdb9326327c3aa3a7389a494d4 WatchSource:0}: Error finding container 5e888d3acbf9145c1b2d22546787ead5f79d27bdb9326327c3aa3a7389a494d4: Status 404 returned error can't find the container with id 5e888d3acbf9145c1b2d22546787ead5f79d27bdb9326327c3aa3a7389a494d4 Oct 29 22:58:05.887416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:05.883619 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:06.472110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:06.472071 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" event={"ID":"f97607e1-3c37-4b30-99ef-e9d2fe142938","Type":"ContainerStarted","Data":"c04cec3636736c3d04d00248186da738e163dde88c373f749af36f51b5dcc24b"} Oct 29 22:58:06.472110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:06.472122 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" event={"ID":"f97607e1-3c37-4b30-99ef-e9d2fe142938","Type":"ContainerStarted","Data":"5e888d3acbf9145c1b2d22546787ead5f79d27bdb9326327c3aa3a7389a494d4"} Oct 29 22:58:06.475435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:06.475374 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:06.508306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:06.508255 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" podStartSLOduration=1.5082413030000001 podStartE2EDuration="1.508241303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:06.507458731 +0000 UTC m=+2548.988776224" watchObservedRunningTime="2026-10-29 22:58:06.508241303 +0000 UTC m=+2548.989558850" Oct 29 22:58:07.677547 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:07.671779 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:08.945866 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:08.942466 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:09.155131 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.155088 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:09.449683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.449647 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms"] Oct 29 22:58:09.451097 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.451075 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7eef245-d0ec-4b6c-a7e1-2a2c4e806800" containerName="istio-proxy" Oct 29 22:58:09.455291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.455272 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:09.481997 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.481970 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6f83c\"" Oct 29 22:58:09.482194 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.482038 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6f83c-dockercfg-xzz64\"" Oct 29 22:58:09.487535 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.487501 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms"] Oct 29 22:58:09.581628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.581592 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm"] Oct 29 22:58:09.587575 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.587551 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:09.590875 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.590853 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423d5fb-4d0b-4357-ad79-9e6877468e56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms\" (UID: \"b423d5fb-4d0b-4357-ad79-9e6877468e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:09.591096 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.591068 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78ss\" (UniqueName: \"kubernetes.io/projected/b423d5fb-4d0b-4357-ad79-9e6877468e56-kube-api-access-l78ss\") pod \"payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms\" (UID: \"b423d5fb-4d0b-4357-ad79-9e6877468e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:09.613095 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.613058 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm"] Oct 29 22:58:09.692949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.692909 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slngr\" (UniqueName: \"kubernetes.io/projected/d701e362-cc92-46b6-a889-86b1d7caab3c-kube-api-access-slngr\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm\" (UID: \"d701e362-cc92-46b6-a889-86b1d7caab3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:09.693191 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.693171 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423d5fb-4d0b-4357-ad79-9e6877468e56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms\" (UID: \"b423d5fb-4d0b-4357-ad79-9e6877468e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:09.693282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.693259 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d701e362-cc92-46b6-a889-86b1d7caab3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm\" (UID: \"d701e362-cc92-46b6-a889-86b1d7caab3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:09.693448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.693420 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78ss\" (UniqueName: \"kubernetes.io/projected/b423d5fb-4d0b-4357-ad79-9e6877468e56-kube-api-access-l78ss\") pod \"payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms\" (UID: \"b423d5fb-4d0b-4357-ad79-9e6877468e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:09.694028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.694008 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423d5fb-4d0b-4357-ad79-9e6877468e56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms\" (UID: \"b423d5fb-4d0b-4357-ad79-9e6877468e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:09.711713 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.711646 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78ss\" (UniqueName: \"kubernetes.io/projected/b423d5fb-4d0b-4357-ad79-9e6877468e56-kube-api-access-l78ss\") pod \"payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms\" (UID: \"b423d5fb-4d0b-4357-ad79-9e6877468e56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:09.773560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.773528 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:09.794649 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.794623 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d701e362-cc92-46b6-a889-86b1d7caab3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm\" (UID: \"d701e362-cc92-46b6-a889-86b1d7caab3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:09.794803 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.794782 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slngr\" (UniqueName: \"kubernetes.io/projected/d701e362-cc92-46b6-a889-86b1d7caab3c-kube-api-access-slngr\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm\" (UID: \"d701e362-cc92-46b6-a889-86b1d7caab3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:09.795600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.795573 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d701e362-cc92-46b6-a889-86b1d7caab3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm\" (UID: \"d701e362-cc92-46b6-a889-86b1d7caab3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:09.812277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.812254 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slngr\" (UniqueName: \"kubernetes.io/projected/d701e362-cc92-46b6-a889-86b1d7caab3c-kube-api-access-slngr\") pod \"payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm\" (UID: \"d701e362-cc92-46b6-a889-86b1d7caab3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:09.906419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:09.905997 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:10.092250 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:58:10.092214 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb423d5fb_4d0b_4357_ad79_9e6877468e56.slice/crio-f441e71dea8aedf6051202f20183d4e3639babb9cbd97c2b289c877c25a45f2f WatchSource:0}: Error finding container f441e71dea8aedf6051202f20183d4e3639babb9cbd97c2b289c877c25a45f2f: Status 404 returned error can't find the container with id f441e71dea8aedf6051202f20183d4e3639babb9cbd97c2b289c877c25a45f2f Oct 29 22:58:10.092720 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.092540 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms"] Oct 29 22:58:10.240203 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.240170 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm"] Oct 29 22:58:10.313432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.313245 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:10.521973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.521883 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" event={"ID":"b423d5fb-4d0b-4357-ad79-9e6877468e56","Type":"ContainerStarted","Data":"5dbb97505a3c32c52576389d4200a12f5d493ee8144a1530a455e3967c4ce2ce"} Oct 29 22:58:10.521973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.521960 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" event={"ID":"b423d5fb-4d0b-4357-ad79-9e6877468e56","Type":"ContainerStarted","Data":"f441e71dea8aedf6051202f20183d4e3639babb9cbd97c2b289c877c25a45f2f"} Oct 29 22:58:10.522169 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.521997 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:10.523751 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.523724 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" event={"ID":"d701e362-cc92-46b6-a889-86b1d7caab3c","Type":"ContainerStarted","Data":"b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af"} Oct 29 22:58:10.523914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.523772 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" event={"ID":"d701e362-cc92-46b6-a889-86b1d7caab3c","Type":"ContainerStarted","Data":"c8d8b4025a3a161b06e99279d8da4998a6b465c75dd4f33d3ced87bd124f9121"} Oct 29 22:58:10.523914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.523801 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:10.568160 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.568101 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" podStartSLOduration=1.568086782 podStartE2EDuration="1.568086782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:10.563219645 +0000 UTC m=+2553.044537139" watchObservedRunningTime="2026-10-29 22:58:10.568086782 +0000 UTC m=+2553.049404274" Oct 29 22:58:10.606747 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:10.605185 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" podStartSLOduration=1.6051680689999999 podStartE2EDuration="1.605168069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:10.601504142 +0000 UTC m=+2553.082821636" watchObservedRunningTime="2026-10-29 22:58:10.605168069 +0000 UTC m=+2553.086485563" Oct 29 22:58:11.794869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:11.794831 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:11.986432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:11.985000 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:13.717120 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:58:13.717078 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:58:13.776422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:13.775809 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:15.475325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:15.475287 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:15.489342 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:15.489316 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:15.539029 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:15.538985 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:15.574565 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:15.574531 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 22:58:17.048815 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.048781 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl"] Oct 29 22:58:17.054028 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.054006 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.078326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.078299 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-bcb810-serving-cert\"" Oct 29 22:58:17.106289 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.105985 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl"] Oct 29 22:58:17.122832 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.122797 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.123119 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.123087 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-metrics-certs\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.123232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.123150 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.123379 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.123342 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2vcb\" (UniqueName: \"kubernetes.io/projected/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-kube-api-access-t2vcb\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.223847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.223812 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:17.224327 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.224299 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.224529 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.224504 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-metrics-certs\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.224597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.224533 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.224670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.224617 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2vcb\" (UniqueName: \"kubernetes.io/projected/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-kube-api-access-t2vcb\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.225177 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.225153 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.227207 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.227181 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-metrics-certs\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.227305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.227259 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.245953 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.245932 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2vcb\" (UniqueName: \"kubernetes.io/projected/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-kube-api-access-t2vcb\") pod \"maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.374196 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.374110 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:17.586157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.586077 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:17.705971 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:17.705941 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl"] Oct 29 22:58:17.706988 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:58:17.706952 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dddf4d0_c8b0_4dc4_83b3_27d23aa54d02.slice/crio-36b64f0a1cd4d4a30e3ae8b42736e499a5ce16abc15d455800cef2d8ccf4b490 WatchSource:0}: Error finding container 36b64f0a1cd4d4a30e3ae8b42736e499a5ce16abc15d455800cef2d8ccf4b490: Status 404 returned error can't find the container with id 36b64f0a1cd4d4a30e3ae8b42736e499a5ce16abc15d455800cef2d8ccf4b490 Oct 29 22:58:18.643467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:18.639338 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" event={"ID":"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02","Type":"ContainerStarted","Data":"c75d860b197105167fd2268f6657785086f4217b98146fe92871568d3d1930c5"} Oct 29 22:58:18.643467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:18.639456 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" event={"ID":"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02","Type":"ContainerStarted","Data":"36b64f0a1cd4d4a30e3ae8b42736e499a5ce16abc15d455800cef2d8ccf4b490"} Oct 29 22:58:18.643467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:18.640746 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:18.681035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:18.680980 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" podStartSLOduration=1.165099523 podStartE2EDuration="1.680964271s" totalImagesPullingTime="515.864748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:18.677762558 +0000 UTC m=+2561.159080052" watchObservedRunningTime="2026-10-29 22:58:18.680964271 +0000 UTC m=+2561.162281764" Oct 29 22:58:19.261038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:19.260958 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:21.126416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:21.123364 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:21.533334 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:21.533302 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 22:58:21.533598 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:21.533378 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 22:58:22.500356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.500320 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:22.680527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.680480 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:22.863560 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.863475 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz"] Oct 29 22:58:22.895786 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.895753 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:22.948040 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.948007 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1dabb8-dockercfg-m759l\"" Oct 29 22:58:22.948378 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.948356 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1dabb8\"" Oct 29 22:58:22.953499 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.953466 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz"] Oct 29 22:58:22.991281 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.991247 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg9bt\" (UniqueName: \"kubernetes.io/projected/cd12124e-237a-49c3-add4-2cfd0f83b589-kube-api-access-mg9bt\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz\" (UID: \"cd12124e-237a-49c3-add4-2cfd0f83b589\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:22.991525 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:22.991505 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd12124e-237a-49c3-add4-2cfd0f83b589-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz\" (UID: \"cd12124e-237a-49c3-add4-2cfd0f83b589\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:23.092836 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:23.092799 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg9bt\" (UniqueName: \"kubernetes.io/projected/cd12124e-237a-49c3-add4-2cfd0f83b589-kube-api-access-mg9bt\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz\" (UID: \"cd12124e-237a-49c3-add4-2cfd0f83b589\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:23.093026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:23.092868 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd12124e-237a-49c3-add4-2cfd0f83b589-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz\" (UID: \"cd12124e-237a-49c3-add4-2cfd0f83b589\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:23.093909 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:23.093882 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd12124e-237a-49c3-add4-2cfd0f83b589-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz\" (UID: \"cd12124e-237a-49c3-add4-2cfd0f83b589\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:23.121900 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:23.121821 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg9bt\" (UniqueName: \"kubernetes.io/projected/cd12124e-237a-49c3-add4-2cfd0f83b589-kube-api-access-mg9bt\") pod \"payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz\" (UID: \"cd12124e-237a-49c3-add4-2cfd0f83b589\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:23.260472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:23.260431 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:23.600611 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:23.600576 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz"] Oct 29 22:58:23.601040 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:58:23.600825 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd12124e_237a_49c3_add4_2cfd0f83b589.slice/crio-97869ca412919502aad8eb91f0a33ce70af4b4602e63d4ac9bb7a0d9ee043e84 WatchSource:0}: Error finding container 97869ca412919502aad8eb91f0a33ce70af4b4602e63d4ac9bb7a0d9ee043e84: Status 404 returned error can't find the container with id 97869ca412919502aad8eb91f0a33ce70af4b4602e63d4ac9bb7a0d9ee043e84 Oct 29 22:58:23.728460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:23.727768 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:23.740317 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:23.740279 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" event={"ID":"cd12124e-237a-49c3-add4-2cfd0f83b589","Type":"ContainerStarted","Data":"97869ca412919502aad8eb91f0a33ce70af4b4602e63d4ac9bb7a0d9ee043e84"} Oct 29 22:58:24.749322 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:24.749286 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" event={"ID":"cd12124e-237a-49c3-add4-2cfd0f83b589","Type":"ContainerStarted","Data":"0b6b6c1ed80080205f6efdfb56a529faa5a7214e9c1a6713326f969e1f1ddc77"} Oct 29 22:58:24.749822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:24.749346 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:24.785735 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:24.785684 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" podStartSLOduration=2.785669548 podStartE2EDuration="2.785669548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:24.784435997 +0000 UTC m=+2567.265753491" watchObservedRunningTime="2026-10-29 22:58:24.785669548 +0000 UTC m=+2567.266987077" Oct 29 22:58:25.450659 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:25.450599 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:25.662338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:25.662309 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 22:58:27.045670 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:27.045633 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:28.612660 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:28.612624 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:30.202989 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:30.202943 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:31.969415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:31.969127 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:32.294022 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:32.293921 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:33.322383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.322348 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:33.365797 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.365761 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf"] Oct 29 22:58:33.366014 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.365990 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" podUID="22dbafd3-5640-415b-b8ab-248aa8079d63" containerName="payload-processing" containerID="cri-o://f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1" gracePeriod=30 Oct 29 22:58:33.399270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.399235 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch"] Oct 29 22:58:33.399552 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.399525 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" podUID="9c999451-dd15-4eac-8946-c5e0e7c060a5" containerName="payload-pre-processing" containerID="cri-o://8398f0b32ebd3778d6239d2f710cf07ee35c8712a176c3a20a05ec886b89d874" gracePeriod=30 Oct 29 22:58:33.495947 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.495913 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:33.634848 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.634643 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:33.955873 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.955557 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:58:33.980908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.978638 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:33.986238 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.986203 5610 generic.go:356] "Generic (PLEG): container finished" podID="22dbafd3-5640-415b-b8ab-248aa8079d63" containerID="f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1" exitCode=0 Oct 29 22:58:33.986417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.986272 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" Oct 29 22:58:33.986417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.986379 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" event={"ID":"22dbafd3-5640-415b-b8ab-248aa8079d63","Type":"ContainerDied","Data":"f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1"} Oct 29 22:58:33.986548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.986439 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf" event={"ID":"22dbafd3-5640-415b-b8ab-248aa8079d63","Type":"ContainerDied","Data":"c2259580918feac044011beb5c6efee1b15bcb6e6ccb0896bc526fe8d909e646"} Oct 29 22:58:33.986548 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.986467 5610 scope.go:122] "RemoveContainer" containerID="f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1" Oct 29 22:58:33.988603 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.988569 5610 generic.go:356] "Generic (PLEG): container finished" podID="9c999451-dd15-4eac-8946-c5e0e7c060a5" containerID="8398f0b32ebd3778d6239d2f710cf07ee35c8712a176c3a20a05ec886b89d874" exitCode=0 Oct 29 22:58:33.988788 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:33.988731 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" event={"ID":"9c999451-dd15-4eac-8946-c5e0e7c060a5","Type":"ContainerDied","Data":"8398f0b32ebd3778d6239d2f710cf07ee35c8712a176c3a20a05ec886b89d874"} Oct 29 22:58:34.006032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.004889 5610 scope.go:122] "RemoveContainer" containerID="f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1" Oct 29 22:58:34.006032 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:58:34.005209 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1\": container with ID starting with f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1 not found: ID does not exist" containerID="f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1" Oct 29 22:58:34.006032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.005248 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1"} err="failed to get container status \"f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1\": rpc error: code = NotFound desc = could not find container \"f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1\": container with ID starting with f40772d1afab0e65b4875507d22e5e8c3fcafda55f6a9eae726598922bc0fdb1 not found: ID does not exist" Oct 29 22:58:34.006032 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.005405 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:58:34.012241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.011465 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dbafd3-5640-415b-b8ab-248aa8079d63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dbafd3-5640-415b-b8ab-248aa8079d63-plugins-config-volume\") pod \"22dbafd3-5640-415b-b8ab-248aa8079d63\" (UID: \"22dbafd3-5640-415b-b8ab-248aa8079d63\") " Oct 29 22:58:34.012241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.011569 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dbafd3-5640-415b-b8ab-248aa8079d63-kube-api-access-jf4ht\" (UniqueName: \"kubernetes.io/projected/22dbafd3-5640-415b-b8ab-248aa8079d63-kube-api-access-jf4ht\") pod \"22dbafd3-5640-415b-b8ab-248aa8079d63\" (UID: \"22dbafd3-5640-415b-b8ab-248aa8079d63\") " Oct 29 22:58:34.012241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.011952 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dbafd3-5640-415b-b8ab-248aa8079d63-plugins-config-volume" pod "22dbafd3-5640-415b-b8ab-248aa8079d63" (UID: "22dbafd3-5640-415b-b8ab-248aa8079d63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:34.012527 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.012276 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22dbafd3-5640-415b-b8ab-248aa8079d63-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:58:34.014746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.014720 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dbafd3-5640-415b-b8ab-248aa8079d63-kube-api-access-jf4ht" pod "22dbafd3-5640-415b-b8ab-248aa8079d63" (UID: "22dbafd3-5640-415b-b8ab-248aa8079d63"). InnerVolumeSpecName "kube-api-access-jf4ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:34.121481 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.121435 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c999451-dd15-4eac-8946-c5e0e7c060a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c999451-dd15-4eac-8946-c5e0e7c060a5-plugins-config-volume\") pod \"9c999451-dd15-4eac-8946-c5e0e7c060a5\" (UID: \"9c999451-dd15-4eac-8946-c5e0e7c060a5\") " Oct 29 22:58:34.121908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.121878 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c999451-dd15-4eac-8946-c5e0e7c060a5-kube-api-access-ld877\" (UniqueName: \"kubernetes.io/projected/9c999451-dd15-4eac-8946-c5e0e7c060a5-kube-api-access-ld877\") pod \"9c999451-dd15-4eac-8946-c5e0e7c060a5\" (UID: \"9c999451-dd15-4eac-8946-c5e0e7c060a5\") " Oct 29 22:58:34.123155 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.123118 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4ht\" (UniqueName: \"kubernetes.io/projected/22dbafd3-5640-415b-b8ab-248aa8079d63-kube-api-access-jf4ht\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:58:34.124173 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.124148 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c999451-dd15-4eac-8946-c5e0e7c060a5-plugins-config-volume" pod "9c999451-dd15-4eac-8946-c5e0e7c060a5" (UID: "9c999451-dd15-4eac-8946-c5e0e7c060a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:34.151466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.141175 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c999451-dd15-4eac-8946-c5e0e7c060a5-kube-api-access-ld877" pod "9c999451-dd15-4eac-8946-c5e0e7c060a5" (UID: "9c999451-dd15-4eac-8946-c5e0e7c060a5"). InnerVolumeSpecName "kube-api-access-ld877". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:34.155148 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.154772 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:34.224261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.224183 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c999451-dd15-4eac-8946-c5e0e7c060a5-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:58:34.224261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.224220 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld877\" (UniqueName: \"kubernetes.io/projected/9c999451-dd15-4eac-8946-c5e0e7c060a5-kube-api-access-ld877\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:58:34.333935 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.333886 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf"] Oct 29 22:58:34.337383 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.337363 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09fc89-847d958946-mcjkf"] Oct 29 22:58:34.548326 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.548249 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:34.998663 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.998626 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" Oct 29 22:58:34.998969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.998734 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch" event={"ID":"9c999451-dd15-4eac-8946-c5e0e7c060a5","Type":"ContainerDied","Data":"91e3bacdd021ca151627818de2e6fe3c77cc305c1636b7e57a82ad3500f37ba6"} Oct 29 22:58:34.998969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:34.998801 5610 scope.go:122] "RemoveContainer" containerID="8398f0b32ebd3778d6239d2f710cf07ee35c8712a176c3a20a05ec886b89d874" Oct 29 22:58:35.046288 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:35.046264 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch"] Oct 29 22:58:35.055448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:35.055425 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09fc89-868c8bb447-dr4ch"] Oct 29 22:58:35.645673 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:35.645630 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dbafd3-5640-415b-b8ab-248aa8079d63" path="/var/lib/kubelet/pods/22dbafd3-5640-415b-b8ab-248aa8079d63/volumes" Oct 29 22:58:35.646221 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:35.646198 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c999451-dd15-4eac-8946-c5e0e7c060a5" path="/var/lib/kubelet/pods/9c999451-dd15-4eac-8946-c5e0e7c060a5/volumes" Oct 29 22:58:35.762347 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:35.762320 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 22:58:35.857566 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:35.857516 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:37.428003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:37.427963 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:39.131404 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.131354 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:39.506427 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.506374 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9"] Oct 29 22:58:39.507345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.507321 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9c999451-dd15-4eac-8946-c5e0e7c060a5" containerName="payload-pre-processing" Oct 29 22:58:39.507345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.507344 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c999451-dd15-4eac-8946-c5e0e7c060a5" containerName="payload-pre-processing" Oct 29 22:58:39.507597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.507380 5610 cpu_manager.go:414] "containerMap: removing container" podUID="22dbafd3-5640-415b-b8ab-248aa8079d63" containerName="payload-processing" Oct 29 22:58:39.507597 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.507387 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dbafd3-5640-415b-b8ab-248aa8079d63" containerName="payload-processing" Oct 29 22:58:39.507708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.507597 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dbafd3-5640-415b-b8ab-248aa8079d63" containerName="payload-processing" Oct 29 22:58:39.507708 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.507631 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c999451-dd15-4eac-8946-c5e0e7c060a5" containerName="payload-pre-processing" Oct 29 22:58:39.511723 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.511703 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.524600 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.524574 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2ae845-openshift-default-dockercfg-5hvm9\"" Oct 29 22:58:39.544553 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.544533 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9"] Oct 29 22:58:39.618758 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.618722 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.618984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.618785 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.618984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.618863 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.618984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.618917 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkf74\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-kube-api-access-dkf74\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.618984 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.618977 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.619242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.619005 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.619242 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.619106 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.619354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.619314 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.619464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.619353 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.720894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.720858 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.720907 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.720930 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.720971 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkf74\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-kube-api-access-dkf74\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.720990 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721014 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721205 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721083 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721329 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721367 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721372 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721669 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721497 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-credential-socket\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721730 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-data\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721784 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-certs\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.721887 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.721812 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istiod-ca-cert\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.723675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.723652 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-envoy\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.723777 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.723765 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-podinfo\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.738899 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.738877 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkf74\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-kube-api-access-dkf74\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.738999 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.738905 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-token\") pod \"e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.827234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.827150 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:39.874051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:39.874012 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:40.185338 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:40.185258 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9"] Oct 29 22:58:40.190593 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:58:40.190563 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c21b381_6e1d_4f1c_9234_554b1f4e0952.slice/crio-0610d4b0042fe0fe5cf93ecacb9808711ef1593b7ae2b45f03578262f4fd2528 WatchSource:0}: Error finding container 0610d4b0042fe0fe5cf93ecacb9808711ef1593b7ae2b45f03578262f4fd2528: Status 404 returned error can't find the container with id 0610d4b0042fe0fe5cf93ecacb9808711ef1593b7ae2b45f03578262f4fd2528 Oct 29 22:58:40.194004 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:40.193970 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:40.194110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:40.194043 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:40.194110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:40.194073 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 22:58:40.335051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:40.334950 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:40.682416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:40.672600 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:40.838810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:40.838772 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:41.057018 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.056982 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" event={"ID":"0c21b381-6e1d-4f1c-9234-554b1f4e0952","Type":"ContainerStarted","Data":"f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b"} Oct 29 22:58:41.057213 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.057048 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" event={"ID":"0c21b381-6e1d-4f1c-9234-554b1f4e0952","Type":"ContainerStarted","Data":"0610d4b0042fe0fe5cf93ecacb9808711ef1593b7ae2b45f03578262f4fd2528"} Oct 29 22:58:41.107847 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.107783 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podStartSLOduration=2.107764999 podStartE2EDuration="2.107764999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:41.105996236 +0000 UTC m=+2583.587313729" watchObservedRunningTime="2026-10-29 22:58:41.107764999 +0000 UTC m=+2583.589082493" Oct 29 22:58:41.141419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.138517 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:41.333741 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.330290 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:41.706452 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.706372 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:41.828144 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.828109 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:41.847587 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.847537 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:58:41.847949 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:41.847884 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 29 22:58:42.050036 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:42.049925 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:42.827486 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:42.827452 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:58:42.828204 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:42.827508 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 29 22:58:43.548822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:43.548779 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:43.727080 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:58:43.727039 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:58:43.827570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:43.827485 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:58:43.827570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:43.827536 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 29 22:58:44.692570 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:44.692528 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:44.829277 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:44.829230 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:58:44.829988 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:44.829957 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 29 22:58:45.008964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:45.008909 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:45.828351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:45.828304 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:58:45.828608 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:45.828384 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 29 22:58:46.229349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.229273 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:46.604822 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.604740 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:46.803903 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.803862 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4"] Oct 29 22:58:46.814435 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.814232 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:46.822039 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.821829 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4"] Oct 29 22:58:46.850428 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.850338 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0906f4\"" Oct 29 22:58:46.851711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.851385 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:58:46.851711 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.851462 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 29 22:58:46.856792 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.855856 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0906f4-dockercfg-xtcfn\"" Oct 29 22:58:46.870970 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:46.870934 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:47.078730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.078323 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z8b8\" (UniqueName: \"kubernetes.io/projected/fa6efccd-b004-4083-a897-6454c274e4fb-kube-api-access-2z8b8\") pod \"payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4\" (UID: \"fa6efccd-b004-4083-a897-6454c274e4fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:47.078730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.078510 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6efccd-b004-4083-a897-6454c274e4fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4\" (UID: \"fa6efccd-b004-4083-a897-6454c274e4fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:47.181473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.180773 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z8b8\" (UniqueName: \"kubernetes.io/projected/fa6efccd-b004-4083-a897-6454c274e4fb-kube-api-access-2z8b8\") pod \"payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4\" (UID: \"fa6efccd-b004-4083-a897-6454c274e4fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:47.181473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.180978 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6efccd-b004-4083-a897-6454c274e4fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4\" (UID: \"fa6efccd-b004-4083-a897-6454c274e4fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:47.205420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.182173 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6efccd-b004-4083-a897-6454c274e4fb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4\" (UID: \"fa6efccd-b004-4083-a897-6454c274e4fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:47.226184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.226119 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z8b8\" (UniqueName: \"kubernetes.io/projected/fa6efccd-b004-4083-a897-6454c274e4fb-kube-api-access-2z8b8\") pod \"payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4\" (UID: \"fa6efccd-b004-4083-a897-6454c274e4fb\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:47.400919 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.400871 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:47.857865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.849738 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 29 22:58:47.857865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:47.849804 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 29 22:58:48.455063 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:58:48.455005 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6efccd_b004_4083_a897_6454c274e4fb.slice/crio-fcaaeabfe66f8891e8c70755a3b0ec3d49d1b671d028a30abcc1fbc55f1c1582 WatchSource:0}: Error finding container fcaaeabfe66f8891e8c70755a3b0ec3d49d1b671d028a30abcc1fbc55f1c1582: Status 404 returned error can't find the container with id fcaaeabfe66f8891e8c70755a3b0ec3d49d1b671d028a30abcc1fbc55f1c1582 Oct 29 22:58:48.465444 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:48.463326 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4"] Oct 29 22:58:48.663110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:48.663064 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:48.832389 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:48.832306 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:48.832829 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:48.832806 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:48.833351 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:48.833329 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 22:58:49.250337 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:49.250296 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" event={"ID":"fa6efccd-b004-4083-a897-6454c274e4fb","Type":"ContainerStarted","Data":"3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58"} Oct 29 22:58:49.250613 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:49.250354 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" event={"ID":"fa6efccd-b004-4083-a897-6454c274e4fb","Type":"ContainerStarted","Data":"fcaaeabfe66f8891e8c70755a3b0ec3d49d1b671d028a30abcc1fbc55f1c1582"} Oct 29 22:58:49.288331 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:49.288267 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" podStartSLOduration=3.28824788 podStartE2EDuration="3.28824788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:49.285622014 +0000 UTC m=+2591.766939511" watchObservedRunningTime="2026-10-29 22:58:49.28824788 +0000 UTC m=+2591.769565379" Oct 29 22:58:50.259345 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:50.259311 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:58:50.544296 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:50.544195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:52.077955 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:52.077915 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:52.648260 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:52.648216 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc"] Oct 29 22:58:52.675314 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:52.675273 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s"] Oct 29 22:58:52.842563 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:52.842525 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:53.053824 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:53.053791 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:53.846420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:53.833536 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:55.698424 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:55.694932 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:57.377892 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:57.377832 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:57.649742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:57.649654 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" podUID="61175f67-8f4d-42a3-91dc-fb62b86cdd46" containerName="payload-processing" containerID="cri-o://4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba" gracePeriod=25 Oct 29 22:58:57.675682 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:57.675646 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" podUID="27e007c1-145b-4e23-92b4-2cf4933c3792" containerName="payload-pre-processing" containerID="cri-o://3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7" gracePeriod=25 Oct 29 22:58:57.829316 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:57.829181 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.019828 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.018581 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.036844 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.036808 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:58:58.124219 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.124190 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:58:58.146223 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.146196 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e007c1-145b-4e23-92b4-2cf4933c3792-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e007c1-145b-4e23-92b4-2cf4933c3792-plugins-config-volume\") pod \"27e007c1-145b-4e23-92b4-2cf4933c3792\" (UID: \"27e007c1-145b-4e23-92b4-2cf4933c3792\") " Oct 29 22:58:58.146441 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.146238 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e007c1-145b-4e23-92b4-2cf4933c3792-kube-api-access-79k7n\" (UniqueName: \"kubernetes.io/projected/27e007c1-145b-4e23-92b4-2cf4933c3792-kube-api-access-79k7n\") pod \"27e007c1-145b-4e23-92b4-2cf4933c3792\" (UID: \"27e007c1-145b-4e23-92b4-2cf4933c3792\") " Oct 29 22:58:58.146591 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.146569 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e007c1-145b-4e23-92b4-2cf4933c3792-plugins-config-volume" pod "27e007c1-145b-4e23-92b4-2cf4933c3792" (UID: "27e007c1-145b-4e23-92b4-2cf4933c3792"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:58.147061 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.147041 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27e007c1-145b-4e23-92b4-2cf4933c3792-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.148368 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.148348 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e007c1-145b-4e23-92b4-2cf4933c3792-kube-api-access-79k7n" pod "27e007c1-145b-4e23-92b4-2cf4933c3792" (UID: "27e007c1-145b-4e23-92b4-2cf4933c3792"). InnerVolumeSpecName "kube-api-access-79k7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:58.247679 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.247641 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61175f67-8f4d-42a3-91dc-fb62b86cdd46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61175f67-8f4d-42a3-91dc-fb62b86cdd46-plugins-config-volume\") pod \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\" (UID: \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\") " Oct 29 22:58:58.247882 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.247846 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61175f67-8f4d-42a3-91dc-fb62b86cdd46-kube-api-access-gtkfg\" (UniqueName: \"kubernetes.io/projected/61175f67-8f4d-42a3-91dc-fb62b86cdd46-kube-api-access-gtkfg\") pod \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\" (UID: \"61175f67-8f4d-42a3-91dc-fb62b86cdd46\") " Oct 29 22:58:58.248038 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.248014 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61175f67-8f4d-42a3-91dc-fb62b86cdd46-plugins-config-volume" pod "61175f67-8f4d-42a3-91dc-fb62b86cdd46" (UID: "61175f67-8f4d-42a3-91dc-fb62b86cdd46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:58:58.248356 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.248337 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79k7n\" (UniqueName: \"kubernetes.io/projected/27e007c1-145b-4e23-92b4-2cf4933c3792-kube-api-access-79k7n\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.248469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.248361 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61175f67-8f4d-42a3-91dc-fb62b86cdd46-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.250111 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.250092 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61175f67-8f4d-42a3-91dc-fb62b86cdd46-kube-api-access-gtkfg" pod "61175f67-8f4d-42a3-91dc-fb62b86cdd46" (UID: "61175f67-8f4d-42a3-91dc-fb62b86cdd46"). InnerVolumeSpecName "kube-api-access-gtkfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:58:58.330622 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.330585 5610 generic.go:356] "Generic (PLEG): container finished" podID="61175f67-8f4d-42a3-91dc-fb62b86cdd46" containerID="4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba" exitCode=0 Oct 29 22:58:58.330795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.330641 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" Oct 29 22:58:58.330795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.330658 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" event={"ID":"61175f67-8f4d-42a3-91dc-fb62b86cdd46","Type":"ContainerDied","Data":"4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba"} Oct 29 22:58:58.330795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.330720 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc" event={"ID":"61175f67-8f4d-42a3-91dc-fb62b86cdd46","Type":"ContainerDied","Data":"481f2a4ceec58ade39e0d448d31f9fbcbeecd664e072d39341d521e2f15d728a"} Oct 29 22:58:58.330795 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.330749 5610 scope.go:122] "RemoveContainer" containerID="4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba" Oct 29 22:58:58.332366 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.332341 5610 generic.go:356] "Generic (PLEG): container finished" podID="27e007c1-145b-4e23-92b4-2cf4933c3792" containerID="3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7" exitCode=0 Oct 29 22:58:58.332500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.332386 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" Oct 29 22:58:58.332500 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.332384 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" event={"ID":"27e007c1-145b-4e23-92b4-2cf4933c3792","Type":"ContainerDied","Data":"3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7"} Oct 29 22:58:58.332585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.332529 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s" event={"ID":"27e007c1-145b-4e23-92b4-2cf4933c3792","Type":"ContainerDied","Data":"cb37c7b376a398a6aab76fefb556c0af6eb0eb4fbb9fd4bbb49da7bc0a1e38b4"} Oct 29 22:58:58.343578 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.343546 5610 scope.go:122] "RemoveContainer" containerID="4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba" Oct 29 22:58:58.343956 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:58:58.343933 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba\": container with ID starting with 4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba not found: ID does not exist" containerID="4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba" Oct 29 22:58:58.344026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.343964 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba"} err="failed to get container status \"4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba\": rpc error: code = NotFound desc = could not find container \"4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba\": container with ID starting with 4236962918fab383d86d459f8fef8588ff02cfac07a50d723524566ba5d9a5ba not found: ID does not exist" Oct 29 22:58:58.344026 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.343980 5610 scope.go:122] "RemoveContainer" containerID="3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7" Oct 29 22:58:58.349492 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.349469 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtkfg\" (UniqueName: \"kubernetes.io/projected/61175f67-8f4d-42a3-91dc-fb62b86cdd46-kube-api-access-gtkfg\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:58:58.354413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.354380 5610 scope.go:122] "RemoveContainer" containerID="3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7" Oct 29 22:58:58.354717 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:58:58.354698 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7\": container with ID starting with 3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7 not found: ID does not exist" containerID="3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7" Oct 29 22:58:58.354780 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.354724 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7"} err="failed to get container status \"3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7\": rpc error: code = NotFound desc = could not find container \"3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7\": container with ID starting with 3621ebdcab982e26f93a4448be577918e9388e40be5095dda60baab17831b4c7 not found: ID does not exist" Oct 29 22:58:58.395306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.395270 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc"] Oct 29 22:58:58.407752 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.407723 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-c7b96a-7dc6895976-54nxc"] Oct 29 22:58:58.481416 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.477773 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s"] Oct 29 22:58:58.496483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.484501 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-c7b96a-6cb5fbc5f5-l7s7s"] Oct 29 22:58:58.653494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.644012 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:58.801686 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:58.797361 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:59.566417 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:59.566325 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:58:59.640104 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:59.640071 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e007c1-145b-4e23-92b4-2cf4933c3792" path="/var/lib/kubelet/pods/27e007c1-145b-4e23-92b4-2cf4933c3792/volumes" Oct 29 22:58:59.640645 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:58:59.640627 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61175f67-8f4d-42a3-91dc-fb62b86cdd46" path="/var/lib/kubelet/pods/61175f67-8f4d-42a3-91dc-fb62b86cdd46/volumes" Oct 29 22:59:01.062051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:01.062009 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:01.304050 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:01.302633 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 22:59:01.715214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:01.715153 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:01.875460 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:01.875415 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:02.039582 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:02.039506 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:02.372630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:02.372529 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:03.123210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:03.123180 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:03.353665 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:03.353632 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:03.824810 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:03.824751 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:04.904967 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:04.904936 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:05.023102 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.023070 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8"] Oct 29 22:59:05.023920 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.023904 5610 cpu_manager.go:414] "containerMap: removing container" podUID="61175f67-8f4d-42a3-91dc-fb62b86cdd46" containerName="payload-processing" Oct 29 22:59:05.023976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.023920 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61175f67-8f4d-42a3-91dc-fb62b86cdd46" containerName="payload-processing" Oct 29 22:59:05.023976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.023940 5610 cpu_manager.go:414] "containerMap: removing container" podUID="27e007c1-145b-4e23-92b4-2cf4933c3792" containerName="payload-pre-processing" Oct 29 22:59:05.023976 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.023944 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e007c1-145b-4e23-92b4-2cf4933c3792" containerName="payload-pre-processing" Oct 29 22:59:05.024092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.024082 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="61175f67-8f4d-42a3-91dc-fb62b86cdd46" containerName="payload-processing" Oct 29 22:59:05.024132 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.024095 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e007c1-145b-4e23-92b4-2cf4933c3792" containerName="payload-pre-processing" Oct 29 22:59:05.029445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.029425 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:05.037719 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.037692 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gt7t\" (UniqueName: \"kubernetes.io/projected/eefa47b1-a8dd-40be-8d66-e411c9408d66-kube-api-access-5gt7t\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8\" (UID: \"eefa47b1-a8dd-40be-8d66-e411c9408d66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:05.037843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.037752 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefa47b1-a8dd-40be-8d66-e411c9408d66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8\" (UID: \"eefa47b1-a8dd-40be-8d66-e411c9408d66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:05.048328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.048305 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8"] Oct 29 22:59:05.054286 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.054265 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-14b0c3\"" Oct 29 22:59:05.054465 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.054450 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-14b0c3-dockercfg-s2hjc\"" Oct 29 22:59:05.071467 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.071442 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:05.280419 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.261291 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefa47b1-a8dd-40be-8d66-e411c9408d66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8\" (UID: \"eefa47b1-a8dd-40be-8d66-e411c9408d66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:05.286250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.286198 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gt7t\" (UniqueName: \"kubernetes.io/projected/eefa47b1-a8dd-40be-8d66-e411c9408d66-kube-api-access-5gt7t\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8\" (UID: \"eefa47b1-a8dd-40be-8d66-e411c9408d66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:05.287172 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.284033 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd"] Oct 29 22:59:05.293343 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.262710 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefa47b1-a8dd-40be-8d66-e411c9408d66-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8\" (UID: \"eefa47b1-a8dd-40be-8d66-e411c9408d66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:05.326908 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.326881 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd"] Oct 29 22:59:05.327261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.327238 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:05.338885 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.338855 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gt7t\" (UniqueName: \"kubernetes.io/projected/eefa47b1-a8dd-40be-8d66-e411c9408d66-kube-api-access-5gt7t\") pod \"payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8\" (UID: \"eefa47b1-a8dd-40be-8d66-e411c9408d66\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:05.344568 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.344539 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:05.388564 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.388527 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363eca02-06d0-4681-b9bb-057ecff62e64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd\" (UID: \"363eca02-06d0-4681-b9bb-057ecff62e64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:05.388706 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.388587 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg225\" (UniqueName: \"kubernetes.io/projected/363eca02-06d0-4681-b9bb-057ecff62e64-kube-api-access-qg225\") pod \"payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd\" (UID: \"363eca02-06d0-4681-b9bb-057ecff62e64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:05.544466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.537119 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363eca02-06d0-4681-b9bb-057ecff62e64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd\" (UID: \"363eca02-06d0-4681-b9bb-057ecff62e64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:05.544466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.537190 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg225\" (UniqueName: \"kubernetes.io/projected/363eca02-06d0-4681-b9bb-057ecff62e64-kube-api-access-qg225\") pod \"payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd\" (UID: \"363eca02-06d0-4681-b9bb-057ecff62e64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:05.544466 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.538577 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363eca02-06d0-4681-b9bb-057ecff62e64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd\" (UID: \"363eca02-06d0-4681-b9bb-057ecff62e64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:05.702414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.679941 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg225\" (UniqueName: \"kubernetes.io/projected/363eca02-06d0-4681-b9bb-057ecff62e64-kube-api-access-qg225\") pod \"payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd\" (UID: \"363eca02-06d0-4681-b9bb-057ecff62e64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:05.702414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.701689 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:05.719374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.719341 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8"] Oct 29 22:59:05.836943 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:05.836823 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:06.038049 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.038017 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd"] Oct 29 22:59:06.038636 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:59:06.038609 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363eca02_06d0_4681_b9bb_057ecff62e64.slice/crio-53d0226873dfb0fc642ba690b501633566ee190d91a121c6cbba44fd9e81d597 WatchSource:0}: Error finding container 53d0226873dfb0fc642ba690b501633566ee190d91a121c6cbba44fd9e81d597: Status 404 returned error can't find the container with id 53d0226873dfb0fc642ba690b501633566ee190d91a121c6cbba44fd9e81d597 Oct 29 22:59:06.442092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.441991 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" event={"ID":"363eca02-06d0-4681-b9bb-057ecff62e64","Type":"ContainerStarted","Data":"6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3"} Oct 29 22:59:06.442092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.442055 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" event={"ID":"363eca02-06d0-4681-b9bb-057ecff62e64","Type":"ContainerStarted","Data":"53d0226873dfb0fc642ba690b501633566ee190d91a121c6cbba44fd9e81d597"} Oct 29 22:59:06.442092 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.442083 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:06.443683 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.443656 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" event={"ID":"eefa47b1-a8dd-40be-8d66-e411c9408d66","Type":"ContainerStarted","Data":"8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a"} Oct 29 22:59:06.443785 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.443700 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" event={"ID":"eefa47b1-a8dd-40be-8d66-e411c9408d66","Type":"ContainerStarted","Data":"cdab8fef9ae4dd382f16669cb8bcc951b1bd43809f94f14918e8a3707eede991"} Oct 29 22:59:06.443785 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.443736 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:06.502206 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.502140 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" podStartSLOduration=1.5021170769999999 podStartE2EDuration="1.502117077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:06.481163048 +0000 UTC m=+2608.962480541" watchObservedRunningTime="2026-10-29 22:59:06.502117077 +0000 UTC m=+2608.983434573" Oct 29 22:59:06.523677 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:06.523624 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" podStartSLOduration=2.523609869 podStartE2EDuration="2.523609869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:06.520804274 +0000 UTC m=+2609.002121767" watchObservedRunningTime="2026-10-29 22:59:06.523609869 +0000 UTC m=+2609.004927361" Oct 29 22:59:07.351109 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:07.351069 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs"] Oct 29 22:59:07.386852 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:07.386818 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m"] Oct 29 22:59:07.552797 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:07.552765 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:07.713325 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:07.713287 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:07.883413 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:07.869089 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:09.368072 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:09.362840 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:09.543291 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:09.543253 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:09.723021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:09.722972 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:10.313349 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:10.313303 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:12.324852 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.324818 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:12.353620 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.353478 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" podUID="3229ddc3-6a63-4629-9ef0-1b57478ddf45" containerName="payload-processing" containerID="cri-o://504c5325e159b5e9fa099ad86545cd5ab3d6b200d04f33e06ff0afa7e35eb06f" gracePeriod=25 Oct 29 22:59:12.394442 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.392284 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" podUID="189a79c6-509e-4db6-bf58-0639630a4e9b" containerName="payload-pre-processing" containerID="cri-o://3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847" gracePeriod=25 Oct 29 22:59:12.512549 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.512513 5610 generic.go:356] "Generic (PLEG): container finished" podID="3229ddc3-6a63-4629-9ef0-1b57478ddf45" containerID="504c5325e159b5e9fa099ad86545cd5ab3d6b200d04f33e06ff0afa7e35eb06f" exitCode=0 Oct 29 22:59:12.512672 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.512583 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" event={"ID":"3229ddc3-6a63-4629-9ef0-1b57478ddf45","Type":"ContainerDied","Data":"504c5325e159b5e9fa099ad86545cd5ab3d6b200d04f33e06ff0afa7e35eb06f"} Oct 29 22:59:12.715250 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.715212 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:59:12.755869 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.755840 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:59:12.801035 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.801001 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189a79c6-509e-4db6-bf58-0639630a4e9b-kube-api-access-v4whm\" (UniqueName: \"kubernetes.io/projected/189a79c6-509e-4db6-bf58-0639630a4e9b-kube-api-access-v4whm\") pod \"189a79c6-509e-4db6-bf58-0639630a4e9b\" (UID: \"189a79c6-509e-4db6-bf58-0639630a4e9b\") " Oct 29 22:59:12.801306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.801286 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189a79c6-509e-4db6-bf58-0639630a4e9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189a79c6-509e-4db6-bf58-0639630a4e9b-plugins-config-volume\") pod \"189a79c6-509e-4db6-bf58-0639630a4e9b\" (UID: \"189a79c6-509e-4db6-bf58-0639630a4e9b\") " Oct 29 22:59:12.801453 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.801431 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3229ddc3-6a63-4629-9ef0-1b57478ddf45-kube-api-access-pscbz\" (UniqueName: \"kubernetes.io/projected/3229ddc3-6a63-4629-9ef0-1b57478ddf45-kube-api-access-pscbz\") pod \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\" (UID: \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\") " Oct 29 22:59:12.809596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.801559 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3229ddc3-6a63-4629-9ef0-1b57478ddf45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3229ddc3-6a63-4629-9ef0-1b57478ddf45-plugins-config-volume\") pod \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\" (UID: \"3229ddc3-6a63-4629-9ef0-1b57478ddf45\") " Oct 29 22:59:12.809596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.803022 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189a79c6-509e-4db6-bf58-0639630a4e9b-plugins-config-volume" pod "189a79c6-509e-4db6-bf58-0639630a4e9b" (UID: "189a79c6-509e-4db6-bf58-0639630a4e9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:12.809596 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.809496 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3229ddc3-6a63-4629-9ef0-1b57478ddf45-plugins-config-volume" pod "3229ddc3-6a63-4629-9ef0-1b57478ddf45" (UID: "3229ddc3-6a63-4629-9ef0-1b57478ddf45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:12.812493 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.811745 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189a79c6-509e-4db6-bf58-0639630a4e9b-kube-api-access-v4whm" pod "189a79c6-509e-4db6-bf58-0639630a4e9b" (UID: "189a79c6-509e-4db6-bf58-0639630a4e9b"). InnerVolumeSpecName "kube-api-access-v4whm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:12.815336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.815295 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3229ddc3-6a63-4629-9ef0-1b57478ddf45-kube-api-access-pscbz" pod "3229ddc3-6a63-4629-9ef0-1b57478ddf45" (UID: "3229ddc3-6a63-4629-9ef0-1b57478ddf45"). InnerVolumeSpecName "kube-api-access-pscbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:12.903894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.903811 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189a79c6-509e-4db6-bf58-0639630a4e9b-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:59:12.903894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.903842 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pscbz\" (UniqueName: \"kubernetes.io/projected/3229ddc3-6a63-4629-9ef0-1b57478ddf45-kube-api-access-pscbz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:59:12.903894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.903852 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3229ddc3-6a63-4629-9ef0-1b57478ddf45-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:59:12.903894 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:12.903863 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4whm\" (UniqueName: \"kubernetes.io/projected/189a79c6-509e-4db6-bf58-0639630a4e9b-kube-api-access-v4whm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:59:13.521184 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.521145 5610 generic.go:356] "Generic (PLEG): container finished" podID="189a79c6-509e-4db6-bf58-0639630a4e9b" containerID="3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847" exitCode=0 Oct 29 22:59:13.521701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.521189 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" Oct 29 22:59:13.521701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.521228 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" event={"ID":"189a79c6-509e-4db6-bf58-0639630a4e9b","Type":"ContainerDied","Data":"3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847"} Oct 29 22:59:13.521701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.521285 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m" event={"ID":"189a79c6-509e-4db6-bf58-0639630a4e9b","Type":"ContainerDied","Data":"40f1be4d8e6d0fe7d4ff3d5501dd658080730b0d38534088e7e5253489803178"} Oct 29 22:59:13.521701 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.521317 5610 scope.go:122] "RemoveContainer" containerID="3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847" Oct 29 22:59:13.522804 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.522790 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" Oct 29 22:59:13.522927 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.522804 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs" event={"ID":"3229ddc3-6a63-4629-9ef0-1b57478ddf45","Type":"ContainerDied","Data":"00dc41fa18939c03781fa95324e5190ad80a995eea124c6744db57fa8a4d6422"} Oct 29 22:59:13.534187 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.534162 5610 scope.go:122] "RemoveContainer" containerID="3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847" Oct 29 22:59:13.534594 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:59:13.534573 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847\": container with ID starting with 3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847 not found: ID does not exist" containerID="3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847" Oct 29 22:59:13.534710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.534604 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847"} err="failed to get container status \"3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847\": rpc error: code = NotFound desc = could not find container \"3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847\": container with ID starting with 3eb138aa59f53c91975686812387b9eb2ca68068a28b5366e4a73c4fd19e6847 not found: ID does not exist" Oct 29 22:59:13.534710 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.534626 5610 scope.go:122] "RemoveContainer" containerID="504c5325e159b5e9fa099ad86545cd5ab3d6b200d04f33e06ff0afa7e35eb06f" Oct 29 22:59:13.573234 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.571165 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs"] Oct 29 22:59:13.578865 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.578833 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bcb810-75ccffb45d-hjkxs"] Oct 29 22:59:13.612801 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.612769 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m"] Oct 29 22:59:13.616514 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.616489 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-d4d57fdbc-qg64m"] Oct 29 22:59:13.637495 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.637468 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189a79c6-509e-4db6-bf58-0639630a4e9b" path="/var/lib/kubelet/pods/189a79c6-509e-4db6-bf58-0639630a4e9b/volumes" Oct 29 22:59:13.637929 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:13.637915 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3229ddc3-6a63-4629-9ef0-1b57478ddf45" path="/var/lib/kubelet/pods/3229ddc3-6a63-4629-9ef0-1b57478ddf45/volumes" Oct 29 22:59:13.711690 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:59:13.711651 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:59:14.152526 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:14.152316 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:15.923136 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:15.923095 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:17.460843 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:17.460809 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 22:59:17.461336 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:17.460901 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 22:59:17.610585 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:17.610549 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:19.694420 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:19.694371 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:20.572806 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:20.572765 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx"] Oct 29 22:59:20.604628 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:20.604596 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9"] Oct 29 22:59:20.768143 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:20.768101 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:20.933473 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:20.933353 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:21.106494 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:21.106447 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:21.750236 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:21.750200 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:22.111310 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:22.111226 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:22.271214 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:22.271176 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:22.752232 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:22.752197 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:23.573422 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:23.572002 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:24.281592 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:24.281549 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:24.537372 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:24.537289 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:25.356211 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:25.356170 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:25.573730 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:25.573651 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" podUID="52bf7424-dda4-459d-b823-26355340ffdc" containerName="payload-processing" containerID="cri-o://56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0" gracePeriod=25 Oct 29 22:59:25.605918 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:25.605856 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" podUID="ab96daa8-8158-4802-8084-81af3befaedf" containerName="payload-pre-processing" containerID="cri-o://f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c" gracePeriod=25 Oct 29 22:59:25.946432 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:25.943437 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:59:25.978635 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:25.978610 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:59:26.047374 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.047344 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bf7424-dda4-459d-b823-26355340ffdc-kube-api-access-c52v9\" (UniqueName: \"kubernetes.io/projected/52bf7424-dda4-459d-b823-26355340ffdc-kube-api-access-c52v9\") pod \"52bf7424-dda4-459d-b823-26355340ffdc\" (UID: \"52bf7424-dda4-459d-b823-26355340ffdc\") " Oct 29 22:59:26.047576 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.047449 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bf7424-dda4-459d-b823-26355340ffdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bf7424-dda4-459d-b823-26355340ffdc-plugins-config-volume\") pod \"52bf7424-dda4-459d-b823-26355340ffdc\" (UID: \"52bf7424-dda4-459d-b823-26355340ffdc\") " Oct 29 22:59:26.047661 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.047639 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab96daa8-8158-4802-8084-81af3befaedf-kube-api-access-glpr5\" (UniqueName: \"kubernetes.io/projected/ab96daa8-8158-4802-8084-81af3befaedf-kube-api-access-glpr5\") pod \"ab96daa8-8158-4802-8084-81af3befaedf\" (UID: \"ab96daa8-8158-4802-8084-81af3befaedf\") " Oct 29 22:59:26.047974 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.047944 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bf7424-dda4-459d-b823-26355340ffdc-plugins-config-volume" pod "52bf7424-dda4-459d-b823-26355340ffdc" (UID: "52bf7424-dda4-459d-b823-26355340ffdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:26.048157 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.048134 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab96daa8-8158-4802-8084-81af3befaedf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab96daa8-8158-4802-8084-81af3befaedf-plugins-config-volume\") pod \"ab96daa8-8158-4802-8084-81af3befaedf\" (UID: \"ab96daa8-8158-4802-8084-81af3befaedf\") " Oct 29 22:59:26.048650 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.048620 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab96daa8-8158-4802-8084-81af3befaedf-plugins-config-volume" pod "ab96daa8-8158-4802-8084-81af3befaedf" (UID: "ab96daa8-8158-4802-8084-81af3befaedf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 22:59:26.049251 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.049220 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab96daa8-8158-4802-8084-81af3befaedf-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:59:26.049357 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.049257 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bf7424-dda4-459d-b823-26355340ffdc-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:59:26.050333 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.050308 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab96daa8-8158-4802-8084-81af3befaedf-kube-api-access-glpr5" pod "ab96daa8-8158-4802-8084-81af3befaedf" (UID: "ab96daa8-8158-4802-8084-81af3befaedf"). InnerVolumeSpecName "kube-api-access-glpr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:26.050472 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.050381 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bf7424-dda4-459d-b823-26355340ffdc-kube-api-access-c52v9" pod "52bf7424-dda4-459d-b823-26355340ffdc" (UID: "52bf7424-dda4-459d-b823-26355340ffdc"). InnerVolumeSpecName "kube-api-access-c52v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 22:59:26.150798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.150722 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c52v9\" (UniqueName: \"kubernetes.io/projected/52bf7424-dda4-459d-b823-26355340ffdc-kube-api-access-c52v9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:59:26.150798 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.150750 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glpr5\" (UniqueName: \"kubernetes.io/projected/ab96daa8-8158-4802-8084-81af3befaedf-kube-api-access-glpr5\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 22:59:26.638476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.638437 5610 generic.go:356] "Generic (PLEG): container finished" podID="ab96daa8-8158-4802-8084-81af3befaedf" containerID="f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c" exitCode=0 Oct 29 22:59:26.638476 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.638473 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" Oct 29 22:59:26.638973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.638513 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" event={"ID":"ab96daa8-8158-4802-8084-81af3befaedf","Type":"ContainerDied","Data":"f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c"} Oct 29 22:59:26.638973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.638573 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9" event={"ID":"ab96daa8-8158-4802-8084-81af3befaedf","Type":"ContainerDied","Data":"8f168360a699491033dcef4096d6e831ddd8c351e56428ccb327829106f620b4"} Oct 29 22:59:26.638973 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.638605 5610 scope.go:122] "RemoveContainer" containerID="f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c" Oct 29 22:59:26.645657 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.645629 5610 generic.go:356] "Generic (PLEG): container finished" podID="52bf7424-dda4-459d-b823-26355340ffdc" containerID="56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0" exitCode=0 Oct 29 22:59:26.645775 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.645696 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" Oct 29 22:59:26.645775 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.645724 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" event={"ID":"52bf7424-dda4-459d-b823-26355340ffdc","Type":"ContainerDied","Data":"56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0"} Oct 29 22:59:26.645870 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.645792 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx" event={"ID":"52bf7424-dda4-459d-b823-26355340ffdc","Type":"ContainerDied","Data":"4e3aabef22851d97c787142fcb866408aed91c73b1308245b1196975ae6b116b"} Oct 29 22:59:26.654192 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.654170 5610 scope.go:122] "RemoveContainer" containerID="f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c" Oct 29 22:59:26.654542 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:59:26.654519 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c\": container with ID starting with f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c not found: ID does not exist" containerID="f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c" Oct 29 22:59:26.654633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.654551 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c"} err="failed to get container status \"f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c\": rpc error: code = NotFound desc = could not find container \"f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c\": container with ID starting with f3ecf529a0635e67636c0ac1d290486f761c5bdb29643a12e6fa44fcc8226a0c not found: ID does not exist" Oct 29 22:59:26.654633 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.654569 5610 scope.go:122] "RemoveContainer" containerID="56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0" Oct 29 22:59:26.665226 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.665208 5610 scope.go:122] "RemoveContainer" containerID="56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0" Oct 29 22:59:26.665530 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:59:26.665509 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0\": container with ID starting with 56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0 not found: ID does not exist" containerID="56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0" Oct 29 22:59:26.665630 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.665536 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0"} err="failed to get container status \"56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0\": rpc error: code = NotFound desc = could not find container \"56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0\": container with ID starting with 56c06391932d62b2f4d9358c7a9156c937f5f8a8dd6f6f2541208257e3752ba0 not found: ID does not exist" Oct 29 22:59:26.698099 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.698066 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9"] Oct 29 22:59:26.707415 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.707363 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-e526ec-dcdcdc5f9-g5fq9"] Oct 29 22:59:26.729898 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.729845 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx"] Oct 29 22:59:26.734051 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.734024 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e526ec-59659cfd46-r84hx"] Oct 29 22:59:26.843448 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:26.843246 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:27.639003 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:27.638966 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bf7424-dda4-459d-b823-26355340ffdc" path="/var/lib/kubelet/pods/52bf7424-dda4-459d-b823-26355340ffdc/volumes" Oct 29 22:59:27.639531 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:27.639480 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab96daa8-8158-4802-8084-81af3befaedf" path="/var/lib/kubelet/pods/ab96daa8-8158-4802-8084-81af3befaedf/volumes" Oct 29 22:59:28.574062 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:28.573995 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:29.244761 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:29.244717 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:29.399255 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:29.399214 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:29.610742 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:29.610499 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:30.065923 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.065883 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:30.077270 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.077236 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx"] Oct 29 22:59:30.078210 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078189 5610 cpu_manager.go:414] "containerMap: removing container" podUID="52bf7424-dda4-459d-b823-26355340ffdc" containerName="payload-processing" Oct 29 22:59:30.078305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078213 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bf7424-dda4-459d-b823-26355340ffdc" containerName="payload-processing" Oct 29 22:59:30.078305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078244 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ab96daa8-8158-4802-8084-81af3befaedf" containerName="payload-pre-processing" Oct 29 22:59:30.078305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078255 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab96daa8-8158-4802-8084-81af3befaedf" containerName="payload-pre-processing" Oct 29 22:59:30.078305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078269 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3229ddc3-6a63-4629-9ef0-1b57478ddf45" containerName="payload-processing" Oct 29 22:59:30.078305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078275 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3229ddc3-6a63-4629-9ef0-1b57478ddf45" containerName="payload-processing" Oct 29 22:59:30.078305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078284 5610 cpu_manager.go:414] "containerMap: removing container" podUID="189a79c6-509e-4db6-bf58-0639630a4e9b" containerName="payload-pre-processing" Oct 29 22:59:30.078305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078290 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189a79c6-509e-4db6-bf58-0639630a4e9b" containerName="payload-pre-processing" Oct 29 22:59:30.078693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078448 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3229ddc3-6a63-4629-9ef0-1b57478ddf45" containerName="payload-processing" Oct 29 22:59:30.078693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078463 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="189a79c6-509e-4db6-bf58-0639630a4e9b" containerName="payload-pre-processing" Oct 29 22:59:30.078693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078486 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bf7424-dda4-459d-b823-26355340ffdc" containerName="payload-processing" Oct 29 22:59:30.078693 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.078498 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab96daa8-8158-4802-8084-81af3befaedf" containerName="payload-pre-processing" Oct 29 22:59:30.084241 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.084220 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.106282 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.106254 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx"] Oct 29 22:59:30.111610 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.111587 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0906f4-serving-cert\"" Oct 29 22:59:30.242091 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.242054 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:30.248911 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.248883 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.249320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.248941 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.249320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.249229 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-metrics-certs\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.249320 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.249266 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7r82\" (UniqueName: \"kubernetes.io/projected/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-kube-api-access-z7r82\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.362469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.356799 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.362469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.356861 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.362469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.357036 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-metrics-certs\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.362469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.357066 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7r82\" (UniqueName: \"kubernetes.io/projected/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-kube-api-access-z7r82\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.368110 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.366496 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.377425 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.375027 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.398464 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.392503 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-metrics-certs\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.404328 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.404302 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:30.415528 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.415492 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7r82\" (UniqueName: \"kubernetes.io/projected/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-kube-api-access-z7r82\") pod \"maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:30.706455 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:30.706421 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:31.018067 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:31.018038 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx"] Oct 29 22:59:31.019581 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:59:31.019553 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dd2ee8_782a_4af7_bc3a_d5afc675e88f.slice/crio-165b06e9eb34d64adff9e665661e331f0c27e294387089a14d08ad7794567972 WatchSource:0}: Error finding container 165b06e9eb34d64adff9e665661e331f0c27e294387089a14d08ad7794567972: Status 404 returned error can't find the container with id 165b06e9eb34d64adff9e665661e331f0c27e294387089a14d08ad7794567972 Oct 29 22:59:31.021133 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:31.021114 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 22:59:31.748995 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:31.748959 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" event={"ID":"86dd2ee8-782a-4af7-bc3a-d5afc675e88f","Type":"ContainerStarted","Data":"590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f"} Oct 29 22:59:31.749353 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:31.749020 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" event={"ID":"86dd2ee8-782a-4af7-bc3a-d5afc675e88f","Type":"ContainerStarted","Data":"165b06e9eb34d64adff9e665661e331f0c27e294387089a14d08ad7794567972"} Oct 29 22:59:31.749353 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:31.749112 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:31.789112 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:31.789053 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" podStartSLOduration=1.171500719 podStartE2EDuration="1.789037678s" totalImagesPullingTime="617.536959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:31.788106596 +0000 UTC m=+2634.269424090" watchObservedRunningTime="2026-10-29 22:59:31.789037678 +0000 UTC m=+2634.270355171" Oct 29 22:59:32.182672 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:32.182579 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:33.766414 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:33.757675 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:35.147889 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:35.147843 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:36.891958 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:36.889240 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:37.126079 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:37.126036 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:37.278294 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:37.278249 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:37.774306 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:37.774274 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 22:59:38.402483 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:38.397981 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:40.198261 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:40.194651 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:40.377093 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:40.377037 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:40.624076 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:40.623996 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6"] Oct 29 22:59:40.814783 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:40.814739 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:40.971580 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:40.971541 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:42.347558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:42.347520 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:43.716505 ip-10-0-133-251 kubenswrapper[5610]: E1029 22:59:43.716469 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 22:59:43.830833 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:43.829458 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:45.522794 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:45.522761 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:45.631469 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:45.631425 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" podUID="d6f75d85-98e3-4f25-ba52-70895e9ac0fc" containerName="payload-processing" containerID="cri-o://959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4" gracePeriod=25 Oct 29 22:59:45.697675 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:45.697630 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:45.852778 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:45.852693 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:47.211005 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:47.210927 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:48.845183 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:48.845147 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" podUID="d6f75d85-98e3-4f25-ba52-70895e9ac0fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 22:59:49.028423 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:49.027670 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:50.333621 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.333573 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:50.497266 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.497232 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:50.682524 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.682443 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n"] Oct 29 22:59:50.705007 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.704977 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:50.712964 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.712931 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n"] Oct 29 22:59:50.718305 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.718272 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bcb810-dockercfg-zzzr2\"" Oct 29 22:59:50.723280 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.723254 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dd49\" (UniqueName: \"kubernetes.io/projected/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-kube-api-access-4dd49\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n\" (UID: \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:50.723594 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.723570 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n\" (UID: \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:50.727914 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.727894 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bcb810\"" Oct 29 22:59:50.960215 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.960162 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dd49\" (UniqueName: \"kubernetes.io/projected/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-kube-api-access-4dd49\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n\" (UID: \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:50.960445 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.960294 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n\" (UID: \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:50.961354 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:50.961325 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n\" (UID: \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:51.039969 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:51.039936 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dd49\" (UniqueName: \"kubernetes.io/projected/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-kube-api-access-4dd49\") pod \"payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n\" (UID: \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:51.343480 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:51.343337 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:51.597381 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:51.569021 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:51.742021 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:51.741993 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n"] Oct 29 22:59:51.742843 ip-10-0-133-251 kubenswrapper[5610]: W1029 22:59:51.742809 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba0bcd7_3a49_4f4a_bfbf_6bfa5e6da1ee.slice/crio-eca2fee5cce4e0513dbf8ea03a381316ab12d7952e4761a543abb0780966763e WatchSource:0}: Error finding container eca2fee5cce4e0513dbf8ea03a381316ab12d7952e4761a543abb0780966763e: Status 404 returned error can't find the container with id eca2fee5cce4e0513dbf8ea03a381316ab12d7952e4761a543abb0780966763e Oct 29 22:59:52.047573 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:52.047541 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" event={"ID":"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee","Type":"ContainerStarted","Data":"bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11"} Oct 29 22:59:52.047746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:52.047592 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" event={"ID":"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee","Type":"ContainerStarted","Data":"eca2fee5cce4e0513dbf8ea03a381316ab12d7952e4761a543abb0780966763e"} Oct 29 22:59:52.047746 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:52.047629 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 22:59:52.085961 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:52.085904 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" podStartSLOduration=2.085886062 podStartE2EDuration="2.085886062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:52.08272528 +0000 UTC m=+2654.564042774" watchObservedRunningTime="2026-10-29 22:59:52.085886062 +0000 UTC m=+2654.567203556" Oct 29 22:59:53.213716 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:53.213674 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:55.225129 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:55.225090 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:56.959437 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:56.953442 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:58.575558 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:58.575516 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 22:59:58.844939 ip-10-0-133-251 kubenswrapper[5610]: I1029 22:59:58.844839 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" podUID="d6f75d85-98e3-4f25-ba52-70895e9ac0fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:00:00.342173 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.342001 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:00.354841 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.354811 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt"] Oct 29 23:00:00.360711 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.360686 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:00.371470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.371438 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8"] Oct 29 23:00:00.383776 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.383168 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:00.389270 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.389240 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt"] Oct 29 23:00:00.413493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.413468 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8"] Oct 29 23:00:00.415919 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.415892 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gz22l\"" Oct 29 23:00:00.534839 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.534801 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkdms\" (UniqueName: \"kubernetes.io/projected/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-kube-api-access-kkdms\") pod \"maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8\" (UID: \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:00.535026 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.534934 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjwd\" (UniqueName: \"kubernetes.io/projected/5ce705b5-e86e-4ca9-919a-ff454c193ce9-kube-api-access-fdjwd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt\" (UID: \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:00.535082 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.535029 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8\" (UID: \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:00.535152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.535138 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce705b5-e86e-4ca9-919a-ff454c193ce9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt\" (UID: \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:00.636363 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.636268 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjwd\" (UniqueName: \"kubernetes.io/projected/5ce705b5-e86e-4ca9-919a-ff454c193ce9-kube-api-access-fdjwd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt\" (UID: \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:00.636363 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.636353 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8\" (UID: \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:00.636642 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.636503 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce705b5-e86e-4ca9-919a-ff454c193ce9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt\" (UID: \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:00.636809 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.636783 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkdms\" (UniqueName: \"kubernetes.io/projected/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-kube-api-access-kkdms\") pod \"maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8\" (UID: \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:00.637180 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.637152 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8\" (UID: \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:00.637367 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.637347 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce705b5-e86e-4ca9-919a-ff454c193ce9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt\" (UID: \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:00.657202 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.657171 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjwd\" (UniqueName: \"kubernetes.io/projected/5ce705b5-e86e-4ca9-919a-ff454c193ce9-kube-api-access-fdjwd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt\" (UID: \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:00.657500 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.657482 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkdms\" (UniqueName: \"kubernetes.io/projected/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-kube-api-access-kkdms\") pod \"maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8\" (UID: \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:00.684807 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.684785 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:00.700189 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.700166 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:00.990870 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:00.990842 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt"] Oct 29 23:00:00.992117 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:00:00.992080 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce705b5_e86e_4ca9_919a_ff454c193ce9.slice/crio-98561152e9e8110b90a0e8db32fc98f83688d68f5350438b8223f732dd795575 WatchSource:0}: Error finding container 98561152e9e8110b90a0e8db32fc98f83688d68f5350438b8223f732dd795575: Status 404 returned error can't find the container with id 98561152e9e8110b90a0e8db32fc98f83688d68f5350438b8223f732dd795575 Oct 29 23:00:01.022359 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:01.022331 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8"] Oct 29 23:00:01.026664 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:00:01.026640 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2aaecce_4ed6_4eb6_83b1_0ab6667f1244.slice/crio-0b374adf088bd5d4c6fd09d088176b84d9df1a528130c29c721e26a5ec00a2a0 WatchSource:0}: Error finding container 0b374adf088bd5d4c6fd09d088176b84d9df1a528130c29c721e26a5ec00a2a0: Status 404 returned error can't find the container with id 0b374adf088bd5d4c6fd09d088176b84d9df1a528130c29c721e26a5ec00a2a0 Oct 29 23:00:01.126783 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:01.126748 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" event={"ID":"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244","Type":"ContainerStarted","Data":"0b374adf088bd5d4c6fd09d088176b84d9df1a528130c29c721e26a5ec00a2a0"} Oct 29 23:00:01.128140 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:01.128115 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" event={"ID":"5ce705b5-e86e-4ca9-919a-ff454c193ce9","Type":"ContainerStarted","Data":"98561152e9e8110b90a0e8db32fc98f83688d68f5350438b8223f732dd795575"} Oct 29 23:00:02.137366 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:02.137331 5610 generic.go:356] "Generic (PLEG): container finished" podID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerID="e29205386ac9f2eb8a43f596e8b2bf56cd7e4dcdf2d8923691ebe17263658240" exitCode=60 Oct 29 23:00:02.137861 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:02.137384 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" event={"ID":"5ce705b5-e86e-4ca9-919a-ff454c193ce9","Type":"ContainerDied","Data":"e29205386ac9f2eb8a43f596e8b2bf56cd7e4dcdf2d8923691ebe17263658240"} Oct 29 23:00:02.137861 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:02.137809 5610 scope.go:122] "RemoveContainer" containerID="e29205386ac9f2eb8a43f596e8b2bf56cd7e4dcdf2d8923691ebe17263658240" Oct 29 23:00:02.139285 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:02.139261 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" event={"ID":"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244","Type":"ContainerStarted","Data":"3fc2b86a964c617207a534ca3b5cd33a5fe827dde86387e8877ae6fb390a3521"} Oct 29 23:00:02.216805 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:02.216760 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" podStartSLOduration=2.216746034 podStartE2EDuration="2.216746034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.214836828 +0000 UTC m=+2664.696154322" watchObservedRunningTime="2026-10-29 23:00:02.216746034 +0000 UTC m=+2664.698063526" Oct 29 23:00:02.243971 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:02.243711 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:03.060117 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:03.060079 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 23:00:03.151747 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:03.151705 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" event={"ID":"5ce705b5-e86e-4ca9-919a-ff454c193ce9","Type":"ContainerStarted","Data":"043402642bc5d77e1d4db5b2799281c76eaa5a2b2ad15cac5514a2bb32eb33a8"} Oct 29 23:00:03.153733 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:03.153701 5610 generic.go:356] "Generic (PLEG): container finished" podID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerID="3fc2b86a964c617207a534ca3b5cd33a5fe827dde86387e8877ae6fb390a3521" exitCode=60 Oct 29 23:00:03.153894 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:03.153777 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" event={"ID":"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244","Type":"ContainerDied","Data":"3fc2b86a964c617207a534ca3b5cd33a5fe827dde86387e8877ae6fb390a3521"} Oct 29 23:00:03.154059 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:03.154042 5610 scope.go:122] "RemoveContainer" containerID="3fc2b86a964c617207a534ca3b5cd33a5fe827dde86387e8877ae6fb390a3521" Oct 29 23:00:03.192909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:03.192852 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" podStartSLOduration=3.192833037 podStartE2EDuration="3.192833037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:03.18768699 +0000 UTC m=+2665.669004484" watchObservedRunningTime="2026-10-29 23:00:03.192833037 +0000 UTC m=+2665.674150531" Oct 29 23:00:03.858419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:03.852270 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:04.164679 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:04.164644 5610 generic.go:356] "Generic (PLEG): container finished" podID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerID="043402642bc5d77e1d4db5b2799281c76eaa5a2b2ad15cac5514a2bb32eb33a8" exitCode=60 Oct 29 23:00:04.165148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:04.164706 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" event={"ID":"5ce705b5-e86e-4ca9-919a-ff454c193ce9","Type":"ContainerDied","Data":"043402642bc5d77e1d4db5b2799281c76eaa5a2b2ad15cac5514a2bb32eb33a8"} Oct 29 23:00:04.165148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:04.164783 5610 scope.go:122] "RemoveContainer" containerID="e29205386ac9f2eb8a43f596e8b2bf56cd7e4dcdf2d8923691ebe17263658240" Oct 29 23:00:04.165283 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:04.165151 5610 scope.go:122] "RemoveContainer" containerID="043402642bc5d77e1d4db5b2799281c76eaa5a2b2ad15cac5514a2bb32eb33a8" Oct 29 23:00:04.165530 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:04.165508 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt_odh-ai-gateway-infra(5ce705b5-e86e-4ca9-919a-ff454c193ce9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" Oct 29 23:00:04.168084 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:04.168057 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" event={"ID":"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244","Type":"ContainerStarted","Data":"4eb6cb5b36ac8fe4517d774bddeda7ed11d4f62f177627f927ec3f4e9fe8fb6f"} Oct 29 23:00:05.197460 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:05.197377 5610 scope.go:122] "RemoveContainer" containerID="043402642bc5d77e1d4db5b2799281c76eaa5a2b2ad15cac5514a2bb32eb33a8" Oct 29 23:00:05.197965 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:05.197768 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt_odh-ai-gateway-infra(5ce705b5-e86e-4ca9-919a-ff454c193ce9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" Oct 29 23:00:05.199602 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:05.199564 5610 generic.go:356] "Generic (PLEG): container finished" podID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerID="4eb6cb5b36ac8fe4517d774bddeda7ed11d4f62f177627f927ec3f4e9fe8fb6f" exitCode=60 Oct 29 23:00:05.199983 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:05.199744 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" event={"ID":"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244","Type":"ContainerDied","Data":"4eb6cb5b36ac8fe4517d774bddeda7ed11d4f62f177627f927ec3f4e9fe8fb6f"} Oct 29 23:00:05.199983 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:05.199906 5610 scope.go:122] "RemoveContainer" containerID="3fc2b86a964c617207a534ca3b5cd33a5fe827dde86387e8877ae6fb390a3521" Oct 29 23:00:05.199983 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:05.199973 5610 scope.go:122] "RemoveContainer" containerID="4eb6cb5b36ac8fe4517d774bddeda7ed11d4f62f177627f927ec3f4e9fe8fb6f" Oct 29 23:00:05.200569 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:05.200437 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8_odh-ai-gateway-infra(b2aaecce-4ed6-4eb6-83b1-0ab6667f1244)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" Oct 29 23:00:05.533472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:05.532219 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:05.953240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:05.953210 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 23:00:06.036602 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.036569 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-plugins-config-volume\") pod \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\" (UID: \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\") " Oct 29 23:00:06.036751 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.036642 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-kube-api-access-6s8cr\" (UniqueName: \"kubernetes.io/projected/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-kube-api-access-6s8cr\") pod \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\" (UID: \"d6f75d85-98e3-4f25-ba52-70895e9ac0fc\") " Oct 29 23:00:06.037006 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.036982 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-plugins-config-volume" pod "d6f75d85-98e3-4f25-ba52-70895e9ac0fc" (UID: "d6f75d85-98e3-4f25-ba52-70895e9ac0fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:06.037333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.037316 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:06.038956 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.038923 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-kube-api-access-6s8cr" pod "d6f75d85-98e3-4f25-ba52-70895e9ac0fc" (UID: "d6f75d85-98e3-4f25-ba52-70895e9ac0fc"). InnerVolumeSpecName "kube-api-access-6s8cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:06.138885 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.138814 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s8cr\" (UniqueName: \"kubernetes.io/projected/d6f75d85-98e3-4f25-ba52-70895e9ac0fc-kube-api-access-6s8cr\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:06.212997 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.212969 5610 scope.go:122] "RemoveContainer" containerID="4eb6cb5b36ac8fe4517d774bddeda7ed11d4f62f177627f927ec3f4e9fe8fb6f" Oct 29 23:00:06.213506 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:06.213455 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8_odh-ai-gateway-infra(b2aaecce-4ed6-4eb6-83b1-0ab6667f1244)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" Oct 29 23:00:06.214249 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.214222 5610 generic.go:356] "Generic (PLEG): container finished" podID="d6f75d85-98e3-4f25-ba52-70895e9ac0fc" containerID="959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4" exitCode=0 Oct 29 23:00:06.214344 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.214262 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" Oct 29 23:00:06.214344 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.214299 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" event={"ID":"d6f75d85-98e3-4f25-ba52-70895e9ac0fc","Type":"ContainerDied","Data":"959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4"} Oct 29 23:00:06.214450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.214348 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6" event={"ID":"d6f75d85-98e3-4f25-ba52-70895e9ac0fc","Type":"ContainerDied","Data":"eb21a48bbb098afe153e3cb04d755df49c145f24519cd401c8a67d799812afd3"} Oct 29 23:00:06.214450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.214368 5610 scope.go:122] "RemoveContainer" containerID="959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4" Oct 29 23:00:06.226468 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.226447 5610 scope.go:122] "RemoveContainer" containerID="959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4" Oct 29 23:00:06.226734 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:06.226716 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4\": container with ID starting with 959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4 not found: ID does not exist" containerID="959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4" Oct 29 23:00:06.226792 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.226741 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4"} err="failed to get container status \"959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4\": rpc error: code = NotFound desc = could not find container \"959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4\": container with ID starting with 959652f3607714e9d6cda208b76cfacdc5a95b6ca05225ae44648f30dc5422c4 not found: ID does not exist" Oct 29 23:00:06.296010 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.295973 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6"] Oct 29 23:00:06.307651 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:06.307619 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-40cf9a-579fb97c7b-krvh6"] Oct 29 23:00:07.127699 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:07.127664 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:07.401720 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:07.401642 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:07.584417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:07.583645 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:07.644273 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:07.644232 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f75d85-98e3-4f25-ba52-70895e9ac0fc" path="/var/lib/kubelet/pods/d6f75d85-98e3-4f25-ba52-70895e9ac0fc/volumes" Oct 29 23:00:07.772226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:07.772183 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:07.934417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:07.930926 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:08.091544 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:08.091454 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:00:08.271562 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:08.271528 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:00:09.045077 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:09.045001 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:09.355350 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:09.355264 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:10.712626 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:10.712563 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:12.453347 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:12.453313 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:13.715147 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:13.715108 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:00:13.978629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:13.978472 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:15.344800 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.344755 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6"] Oct 29 23:00:15.346811 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.346778 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d6f75d85-98e3-4f25-ba52-70895e9ac0fc" containerName="payload-processing" Oct 29 23:00:15.346811 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.346809 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f75d85-98e3-4f25-ba52-70895e9ac0fc" containerName="payload-processing" Oct 29 23:00:15.347060 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.347043 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f75d85-98e3-4f25-ba52-70895e9ac0fc" containerName="payload-processing" Oct 29 23:00:15.361448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.352741 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.428865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.403044 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.428865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.403133 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rssr\" (UniqueName: \"kubernetes.io/projected/37858cc2-c0c8-4d60-b15c-53f65745d14e-kube-api-access-2rssr\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.428865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.403305 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.428865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.422440 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6"] Oct 29 23:00:15.474432 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.461619 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 29 23:00:15.544680 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.525581 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.544680 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.525800 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.544680 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.525881 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rssr\" (UniqueName: \"kubernetes.io/projected/37858cc2-c0c8-4d60-b15c-53f65745d14e-kube-api-access-2rssr\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.544680 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.527068 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.601163 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.601011 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.607508 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.603602 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rssr\" (UniqueName: \"kubernetes.io/projected/37858cc2-c0c8-4d60-b15c-53f65745d14e-kube-api-access-2rssr\") pod \"maas-api-revoke-sub-3b858176-c85h6\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.738937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.738906 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:15.863629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:15.863545 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:16.038056 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:16.038018 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:16.055355 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:16.055324 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6"] Oct 29 23:00:16.057072 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:00:16.057047 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37858cc2_c0c8_4d60_b15c_53f65745d14e.slice/crio-7466362c70b6c66ee27f681e1cdba20835974775730bed4821b9e6acf492d48c WatchSource:0}: Error finding container 7466362c70b6c66ee27f681e1cdba20835974775730bed4821b9e6acf492d48c: Status 404 returned error can't find the container with id 7466362c70b6c66ee27f681e1cdba20835974775730bed4821b9e6acf492d48c Oct 29 23:00:16.223211 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:16.223114 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:16.329266 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:16.329220 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" event={"ID":"37858cc2-c0c8-4d60-b15c-53f65745d14e","Type":"ContainerStarted","Data":"7466362c70b6c66ee27f681e1cdba20835974775730bed4821b9e6acf492d48c"} Oct 29 23:00:16.629175 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:16.629137 5610 scope.go:122] "RemoveContainer" containerID="043402642bc5d77e1d4db5b2799281c76eaa5a2b2ad15cac5514a2bb32eb33a8" Oct 29 23:00:16.813445 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:16.813300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:17.038795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:17.038758 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4"] Oct 29 23:00:17.240756 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:17.240722 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:17.362468 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:17.362426 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" event={"ID":"5ce705b5-e86e-4ca9-919a-ff454c193ce9","Type":"ContainerStarted","Data":"d9179ee5e80e77bca8c09bdd26d946c7f36dc4e2fe196fee8fd9d1c7eaa1d2b6"} Oct 29 23:00:17.366739 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:17.366676 5610 generic.go:356] "Generic (PLEG): container finished" podID="37858cc2-c0c8-4d60-b15c-53f65745d14e" containerID="93f3311c5ca5779afee2dfb5c7693854f5aec0e0ebc3e38d5ea6c7bb58ab20da" exitCode=0 Oct 29 23:00:17.367948 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:17.367185 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" event={"ID":"37858cc2-c0c8-4d60-b15c-53f65745d14e","Type":"ContainerDied","Data":"93f3311c5ca5779afee2dfb5c7693854f5aec0e0ebc3e38d5ea6c7bb58ab20da"} Oct 29 23:00:17.427029 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:17.426992 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:17.635935 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:17.635850 5610 scope.go:122] "RemoveContainer" containerID="4eb6cb5b36ac8fe4517d774bddeda7ed11d4f62f177627f927ec3f4e9fe8fb6f" Oct 29 23:00:18.338673 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.338581 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:18.376719 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.376689 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" event={"ID":"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244","Type":"ContainerStarted","Data":"eb123d4e6546c31f2b26acdffd3f883c50856c8fd4362e05c016e01121c88dc7"} Oct 29 23:00:18.378762 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.378735 5610 generic.go:356] "Generic (PLEG): container finished" podID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerID="d9179ee5e80e77bca8c09bdd26d946c7f36dc4e2fe196fee8fd9d1c7eaa1d2b6" exitCode=60 Oct 29 23:00:18.378953 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.378807 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" event={"ID":"5ce705b5-e86e-4ca9-919a-ff454c193ce9","Type":"ContainerDied","Data":"d9179ee5e80e77bca8c09bdd26d946c7f36dc4e2fe196fee8fd9d1c7eaa1d2b6"} Oct 29 23:00:18.378953 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.378867 5610 scope.go:122] "RemoveContainer" containerID="043402642bc5d77e1d4db5b2799281c76eaa5a2b2ad15cac5514a2bb32eb33a8" Oct 29 23:00:18.379320 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.379298 5610 scope.go:122] "RemoveContainer" containerID="d9179ee5e80e77bca8c09bdd26d946c7f36dc4e2fe196fee8fd9d1c7eaa1d2b6" Oct 29 23:00:18.379991 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:18.379704 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt_odh-ai-gateway-infra(5ce705b5-e86e-4ca9-919a-ff454c193ce9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" Oct 29 23:00:18.430967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.430932 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt"] Oct 29 23:00:18.563171 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.563142 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:18.626900 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.626820 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37858cc2-c0c8-4d60-b15c-53f65745d14e-kube-api-access-2rssr\" (UniqueName: \"kubernetes.io/projected/37858cc2-c0c8-4d60-b15c-53f65745d14e-kube-api-access-2rssr\") pod \"37858cc2-c0c8-4d60-b15c-53f65745d14e\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " Oct 29 23:00:18.627086 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.626898 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-xks-ca\") pod \"37858cc2-c0c8-4d60-b15c-53f65745d14e\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " Oct 29 23:00:18.627215 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.627198 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-openshift-ca\") pod \"37858cc2-c0c8-4d60-b15c-53f65745d14e\" (UID: \"37858cc2-c0c8-4d60-b15c-53f65745d14e\") " Oct 29 23:00:18.627622 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.627594 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-openshift-ca" pod "37858cc2-c0c8-4d60-b15c-53f65745d14e" (UID: "37858cc2-c0c8-4d60-b15c-53f65745d14e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:18.627784 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.627765 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-openshift-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.629054 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.629028 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37858cc2-c0c8-4d60-b15c-53f65745d14e-kube-api-access-2rssr" pod "37858cc2-c0c8-4d60-b15c-53f65745d14e" (UID: "37858cc2-c0c8-4d60-b15c-53f65745d14e"). InnerVolumeSpecName "kube-api-access-2rssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:18.629142 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.629064 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-xks-ca" pod "37858cc2-c0c8-4d60-b15c-53f65745d14e" (UID: "37858cc2-c0c8-4d60-b15c-53f65745d14e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:00:18.729430 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.729358 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rssr\" (UniqueName: \"kubernetes.io/projected/37858cc2-c0c8-4d60-b15c-53f65745d14e-kube-api-access-2rssr\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:18.729430 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:18.729413 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37858cc2-c0c8-4d60-b15c-53f65745d14e-maas-api-xks-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.409925 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.409604 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" Oct 29 23:00:19.409925 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.409619 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6" event={"ID":"37858cc2-c0c8-4d60-b15c-53f65745d14e","Type":"ContainerDied","Data":"7466362c70b6c66ee27f681e1cdba20835974775730bed4821b9e6acf492d48c"} Oct 29 23:00:19.409925 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.409714 5610 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7466362c70b6c66ee27f681e1cdba20835974775730bed4821b9e6acf492d48c" Oct 29 23:00:19.419746 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.419716 5610 generic.go:356] "Generic (PLEG): container finished" podID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerID="eb123d4e6546c31f2b26acdffd3f883c50856c8fd4362e05c016e01121c88dc7" exitCode=60 Oct 29 23:00:19.419871 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.419794 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" event={"ID":"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244","Type":"ContainerDied","Data":"eb123d4e6546c31f2b26acdffd3f883c50856c8fd4362e05c016e01121c88dc7"} Oct 29 23:00:19.419927 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.419884 5610 scope.go:122] "RemoveContainer" containerID="4eb6cb5b36ac8fe4517d774bddeda7ed11d4f62f177627f927ec3f4e9fe8fb6f" Oct 29 23:00:19.420309 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.420288 5610 scope.go:122] "RemoveContainer" containerID="eb123d4e6546c31f2b26acdffd3f883c50856c8fd4362e05c016e01121c88dc7" Oct 29 23:00:19.420706 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:19.420681 5610 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8_odh-ai-gateway-infra(b2aaecce-4ed6-4eb6-83b1-0ab6667f1244)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" Oct 29 23:00:19.475217 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.475176 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8"] Oct 29 23:00:19.632092 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.632062 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:19.747013 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.746890 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce705b5-e86e-4ca9-919a-ff454c193ce9-kube-api-access-fdjwd\" (UniqueName: \"kubernetes.io/projected/5ce705b5-e86e-4ca9-919a-ff454c193ce9-kube-api-access-fdjwd\") pod \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\" (UID: \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\") " Oct 29 23:00:19.747496 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.747217 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce705b5-e86e-4ca9-919a-ff454c193ce9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce705b5-e86e-4ca9-919a-ff454c193ce9-maas-api-service-ca\") pod \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\" (UID: \"5ce705b5-e86e-4ca9-919a-ff454c193ce9\") " Oct 29 23:00:19.747634 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.747612 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce705b5-e86e-4ca9-919a-ff454c193ce9-maas-api-service-ca" pod "5ce705b5-e86e-4ca9-919a-ff454c193ce9" (UID: "5ce705b5-e86e-4ca9-919a-ff454c193ce9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:19.747871 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.747852 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce705b5-e86e-4ca9-919a-ff454c193ce9-maas-api-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:19.749354 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.749330 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce705b5-e86e-4ca9-919a-ff454c193ce9-kube-api-access-fdjwd" pod "5ce705b5-e86e-4ca9-919a-ff454c193ce9" (UID: "5ce705b5-e86e-4ca9-919a-ff454c193ce9"). InnerVolumeSpecName "kube-api-access-fdjwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:19.849340 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:19.849293 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdjwd\" (UniqueName: \"kubernetes.io/projected/5ce705b5-e86e-4ca9-919a-ff454c193ce9-kube-api-access-fdjwd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.244427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.243950 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:20.431077 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.431048 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" event={"ID":"5ce705b5-e86e-4ca9-919a-ff454c193ce9","Type":"ContainerDied","Data":"98561152e9e8110b90a0e8db32fc98f83688d68f5350438b8223f732dd795575"} Oct 29 23:00:20.431239 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.431113 5610 scope.go:122] "RemoveContainer" containerID="d9179ee5e80e77bca8c09bdd26d946c7f36dc4e2fe196fee8fd9d1c7eaa1d2b6" Oct 29 23:00:20.431239 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.431216 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt" Oct 29 23:00:20.508284 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.508243 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt"] Oct 29 23:00:20.518483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.518448 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-dbb507-29888580-m8gmt"] Oct 29 23:00:20.621777 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.621749 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:20.781668 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.781550 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-kube-api-access-kkdms\" (UniqueName: \"kubernetes.io/projected/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-kube-api-access-kkdms\") pod \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\" (UID: \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\") " Oct 29 23:00:20.782175 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.781667 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-maas-api-service-ca\") pod \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\" (UID: \"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244\") " Oct 29 23:00:20.782175 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.782089 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-maas-api-service-ca" pod "b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" (UID: "b2aaecce-4ed6-4eb6-83b1-0ab6667f1244"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:20.782527 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.782503 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-maas-api-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:20.784539 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.784509 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-kube-api-access-kkdms" pod "b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" (UID: "b2aaecce-4ed6-4eb6-83b1-0ab6667f1244"). InnerVolumeSpecName "kube-api-access-kkdms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:20.884379 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:20.884341 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkdms\" (UniqueName: \"kubernetes.io/projected/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244-kube-api-access-kkdms\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:21.440671 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:21.440642 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" Oct 29 23:00:21.440847 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:21.440636 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8" event={"ID":"b2aaecce-4ed6-4eb6-83b1-0ab6667f1244","Type":"ContainerDied","Data":"0b374adf088bd5d4c6fd09d088176b84d9df1a528130c29c721e26a5ec00a2a0"} Oct 29 23:00:21.440847 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:21.440785 5610 scope.go:122] "RemoveContainer" containerID="eb123d4e6546c31f2b26acdffd3f883c50856c8fd4362e05c016e01121c88dc7" Oct 29 23:00:21.489041 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:21.489012 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8"] Oct 29 23:00:21.503072 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:21.503038 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0906f4-29888580-7stn8"] Oct 29 23:00:21.638696 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:21.638664 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" path="/var/lib/kubelet/pods/5ce705b5-e86e-4ca9-919a-ff454c193ce9/volumes" Oct 29 23:00:21.639184 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:21.639170 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" path="/var/lib/kubelet/pods/b2aaecce-4ed6-4eb6-83b1-0ab6667f1244/volumes" Oct 29 23:00:21.882924 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:21.882578 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:22.367489 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:22.367437 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" podUID="fa6efccd-b004-4083-a897-6454c274e4fb" containerName="payload-processing" containerID="cri-o://3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58" gracePeriod=25 Oct 29 23:00:22.677213 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:22.677183 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 23:00:22.711501 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:22.711471 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6efccd-b004-4083-a897-6454c274e4fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6efccd-b004-4083-a897-6454c274e4fb-plugins-config-volume\") pod \"fa6efccd-b004-4083-a897-6454c274e4fb\" (UID: \"fa6efccd-b004-4083-a897-6454c274e4fb\") " Oct 29 23:00:22.711697 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:22.711681 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6efccd-b004-4083-a897-6454c274e4fb-kube-api-access-2z8b8\" (UniqueName: \"kubernetes.io/projected/fa6efccd-b004-4083-a897-6454c274e4fb-kube-api-access-2z8b8\") pod \"fa6efccd-b004-4083-a897-6454c274e4fb\" (UID: \"fa6efccd-b004-4083-a897-6454c274e4fb\") " Oct 29 23:00:22.711873 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:22.711846 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6efccd-b004-4083-a897-6454c274e4fb-plugins-config-volume" pod "fa6efccd-b004-4083-a897-6454c274e4fb" (UID: "fa6efccd-b004-4083-a897-6454c274e4fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:22.712325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:22.712301 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6efccd-b004-4083-a897-6454c274e4fb-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:22.714068 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:22.714037 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6efccd-b004-4083-a897-6454c274e4fb-kube-api-access-2z8b8" pod "fa6efccd-b004-4083-a897-6454c274e4fb" (UID: "fa6efccd-b004-4083-a897-6454c274e4fb"). InnerVolumeSpecName "kube-api-access-2z8b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:22.813276 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:22.813246 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z8b8\" (UniqueName: \"kubernetes.io/projected/fa6efccd-b004-4083-a897-6454c274e4fb-kube-api-access-2z8b8\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:23.289999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.289961 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:23.327842 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.327808 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms"] Oct 29 23:00:23.328116 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.328068 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" podUID="b423d5fb-4d0b-4357-ad79-9e6877468e56" containerName="payload-processing" containerID="cri-o://5dbb97505a3c32c52576389d4200a12f5d493ee8144a1530a455e3967c4ce2ce" gracePeriod=30 Oct 29 23:00:23.362883 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.362846 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm"] Oct 29 23:00:23.363214 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.363129 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" podUID="d701e362-cc92-46b6-a889-86b1d7caab3c" containerName="payload-pre-processing" containerID="cri-o://b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af" gracePeriod=30 Oct 29 23:00:23.467159 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.467115 5610 generic.go:356] "Generic (PLEG): container finished" podID="b423d5fb-4d0b-4357-ad79-9e6877468e56" containerID="5dbb97505a3c32c52576389d4200a12f5d493ee8144a1530a455e3967c4ce2ce" exitCode=0 Oct 29 23:00:23.467423 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.467380 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" event={"ID":"b423d5fb-4d0b-4357-ad79-9e6877468e56","Type":"ContainerDied","Data":"5dbb97505a3c32c52576389d4200a12f5d493ee8144a1530a455e3967c4ce2ce"} Oct 29 23:00:23.470116 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.470077 5610 generic.go:356] "Generic (PLEG): container finished" podID="fa6efccd-b004-4083-a897-6454c274e4fb" containerID="3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58" exitCode=0 Oct 29 23:00:23.470499 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.470186 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" event={"ID":"fa6efccd-b004-4083-a897-6454c274e4fb","Type":"ContainerDied","Data":"3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58"} Oct 29 23:00:23.470499 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.470232 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" event={"ID":"fa6efccd-b004-4083-a897-6454c274e4fb","Type":"ContainerDied","Data":"fcaaeabfe66f8891e8c70755a3b0ec3d49d1b671d028a30abcc1fbc55f1c1582"} Oct 29 23:00:23.470499 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.470261 5610 scope.go:122] "RemoveContainer" containerID="3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58" Oct 29 23:00:23.470790 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.470521 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4" Oct 29 23:00:23.495817 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.495016 5610 scope.go:122] "RemoveContainer" containerID="3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58" Oct 29 23:00:23.495817 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:23.495376 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58\": container with ID starting with 3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58 not found: ID does not exist" containerID="3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58" Oct 29 23:00:23.495817 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.495789 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58"} err="failed to get container status \"3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58\": rpc error: code = NotFound desc = could not find container \"3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58\": container with ID starting with 3aaa46c214177db1a523707ee0108ba0b8a53c0a417dc5f05a749ec6b9b23c58 not found: ID does not exist" Oct 29 23:00:23.501691 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.501630 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:23.525055 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.525005 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4"] Oct 29 23:00:23.534863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.534748 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-76b779f4dd-xq6x4"] Oct 29 23:00:23.658414 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.649576 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6efccd-b004-4083-a897-6454c274e4fb" path="/var/lib/kubelet/pods/fa6efccd-b004-4083-a897-6454c274e4fb/volumes" Oct 29 23:00:23.677584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:23.659549 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:24.012614 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.011819 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:24.037177 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.034269 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 23:00:24.075418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.075370 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 23:00:24.133442 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.133014 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b423d5fb-4d0b-4357-ad79-9e6877468e56-kube-api-access-l78ss\" (UniqueName: \"kubernetes.io/projected/b423d5fb-4d0b-4357-ad79-9e6877468e56-kube-api-access-l78ss\") pod \"b423d5fb-4d0b-4357-ad79-9e6877468e56\" (UID: \"b423d5fb-4d0b-4357-ad79-9e6877468e56\") " Oct 29 23:00:24.133442 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.133232 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d701e362-cc92-46b6-a889-86b1d7caab3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d701e362-cc92-46b6-a889-86b1d7caab3c-plugins-config-volume\") pod \"d701e362-cc92-46b6-a889-86b1d7caab3c\" (UID: \"d701e362-cc92-46b6-a889-86b1d7caab3c\") " Oct 29 23:00:24.133696 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.133616 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d701e362-cc92-46b6-a889-86b1d7caab3c-plugins-config-volume" pod "d701e362-cc92-46b6-a889-86b1d7caab3c" (UID: "d701e362-cc92-46b6-a889-86b1d7caab3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:24.133880 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.133697 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d701e362-cc92-46b6-a889-86b1d7caab3c-kube-api-access-slngr\" (UniqueName: \"kubernetes.io/projected/d701e362-cc92-46b6-a889-86b1d7caab3c-kube-api-access-slngr\") pod \"d701e362-cc92-46b6-a889-86b1d7caab3c\" (UID: \"d701e362-cc92-46b6-a889-86b1d7caab3c\") " Oct 29 23:00:24.133880 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.133782 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b423d5fb-4d0b-4357-ad79-9e6877468e56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423d5fb-4d0b-4357-ad79-9e6877468e56-plugins-config-volume\") pod \"b423d5fb-4d0b-4357-ad79-9e6877468e56\" (UID: \"b423d5fb-4d0b-4357-ad79-9e6877468e56\") " Oct 29 23:00:24.134860 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.134464 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d701e362-cc92-46b6-a889-86b1d7caab3c-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:24.134860 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.134824 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b423d5fb-4d0b-4357-ad79-9e6877468e56-plugins-config-volume" pod "b423d5fb-4d0b-4357-ad79-9e6877468e56" (UID: "b423d5fb-4d0b-4357-ad79-9e6877468e56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:24.136153 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.136129 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d701e362-cc92-46b6-a889-86b1d7caab3c-kube-api-access-slngr" pod "d701e362-cc92-46b6-a889-86b1d7caab3c" (UID: "d701e362-cc92-46b6-a889-86b1d7caab3c"). InnerVolumeSpecName "kube-api-access-slngr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:24.141646 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.141616 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b423d5fb-4d0b-4357-ad79-9e6877468e56-kube-api-access-l78ss" pod "b423d5fb-4d0b-4357-ad79-9e6877468e56" (UID: "b423d5fb-4d0b-4357-ad79-9e6877468e56"). InnerVolumeSpecName "kube-api-access-l78ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:24.179471 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.179428 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:24.236314 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.236278 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slngr\" (UniqueName: \"kubernetes.io/projected/d701e362-cc92-46b6-a889-86b1d7caab3c-kube-api-access-slngr\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:24.236314 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.236309 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423d5fb-4d0b-4357-ad79-9e6877468e56-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:24.236314 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.236319 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l78ss\" (UniqueName: \"kubernetes.io/projected/b423d5fb-4d0b-4357-ad79-9e6877468e56-kube-api-access-l78ss\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:24.344677 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.344515 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:24.480175 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.480141 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" Oct 29 23:00:24.480175 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.480167 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms" event={"ID":"b423d5fb-4d0b-4357-ad79-9e6877468e56","Type":"ContainerDied","Data":"f441e71dea8aedf6051202f20183d4e3639babb9cbd97c2b289c877c25a45f2f"} Oct 29 23:00:24.480525 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.480230 5610 scope.go:122] "RemoveContainer" containerID="5dbb97505a3c32c52576389d4200a12f5d493ee8144a1530a455e3967c4ce2ce" Oct 29 23:00:24.484742 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.484716 5610 generic.go:356] "Generic (PLEG): container finished" podID="d701e362-cc92-46b6-a889-86b1d7caab3c" containerID="b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af" exitCode=0 Oct 29 23:00:24.484872 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.484767 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" Oct 29 23:00:24.484872 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.484809 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" event={"ID":"d701e362-cc92-46b6-a889-86b1d7caab3c","Type":"ContainerDied","Data":"b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af"} Oct 29 23:00:24.484988 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.484863 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm" event={"ID":"d701e362-cc92-46b6-a889-86b1d7caab3c","Type":"ContainerDied","Data":"c8d8b4025a3a161b06e99279d8da4998a6b465c75dd4f33d3ced87bd124f9121"} Oct 29 23:00:24.496861 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.496649 5610 scope.go:122] "RemoveContainer" containerID="b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af" Oct 29 23:00:24.509060 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.509041 5610 scope.go:122] "RemoveContainer" containerID="b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af" Oct 29 23:00:24.509365 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:24.509339 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af\": container with ID starting with b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af not found: ID does not exist" containerID="b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af" Oct 29 23:00:24.509493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.509376 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af"} err="failed to get container status \"b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af\": rpc error: code = NotFound desc = could not find container \"b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af\": container with ID starting with b0b6408082466b5e12fdcfc8b988556574ade36df5c5e95ed4fffc7fe6b237af not found: ID does not exist" Oct 29 23:00:24.545070 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.545043 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm"] Oct 29 23:00:24.554601 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.554573 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6f83c-6f8d567c95-zwjhm"] Oct 29 23:00:24.578608 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.578574 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms"] Oct 29 23:00:24.582019 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.581994 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6f83c-7bdcffb664-wcfms"] Oct 29 23:00:24.698894 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:24.698860 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:25.637639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:25.637595 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b423d5fb-4d0b-4357-ad79-9e6877468e56" path="/var/lib/kubelet/pods/b423d5fb-4d0b-4357-ad79-9e6877468e56/volumes" Oct 29 23:00:25.638150 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:25.638109 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d701e362-cc92-46b6-a889-86b1d7caab3c" path="/var/lib/kubelet/pods/d701e362-cc92-46b6-a889-86b1d7caab3c/volumes" Oct 29 23:00:26.048026 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:26.042124 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:27.771411 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:27.771283 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:29.291937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:29.291900 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:29.578468 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:29.578367 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:30.255435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:30.255380 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:30.715672 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:30.715637 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd"] Oct 29 23:00:30.752784 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:30.752749 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8"] Oct 29 23:00:30.936148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:30.936108 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:31.116627 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.116549 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:31.288134 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.288101 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:31.659315 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.659268 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l"] Oct 29 23:00:31.660762 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.660734 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:31.660935 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.660922 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:31.661043 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661018 5610 cpu_manager.go:414] "containerMap: removing container" podUID="fa6efccd-b004-4083-a897-6454c274e4fb" containerName="payload-processing" Oct 29 23:00:31.661165 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661152 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6efccd-b004-4083-a897-6454c274e4fb" containerName="payload-processing" Oct 29 23:00:31.661298 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661287 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.661463 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661437 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.661642 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661613 5610 cpu_manager.go:414] "containerMap: removing container" podUID="37858cc2-c0c8-4d60-b15c-53f65745d14e" containerName="revoke-keys" Oct 29 23:00:31.661725 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661714 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37858cc2-c0c8-4d60-b15c-53f65745d14e" containerName="revoke-keys" Oct 29 23:00:31.661823 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661815 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:31.661895 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661885 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:31.661982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.661973 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.662055 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.662046 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.662143 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.662134 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b423d5fb-4d0b-4357-ad79-9e6877468e56" containerName="payload-processing" Oct 29 23:00:31.662218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.662209 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b423d5fb-4d0b-4357-ad79-9e6877468e56" containerName="payload-processing" Oct 29 23:00:31.662484 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.662468 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.662583 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.662571 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.662681 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.662670 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d701e362-cc92-46b6-a889-86b1d7caab3c" containerName="payload-pre-processing" Oct 29 23:00:31.663682 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.663661 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d701e362-cc92-46b6-a889-86b1d7caab3c" containerName="payload-pre-processing" Oct 29 23:00:31.664040 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664025 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:31.664158 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664147 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.664238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664229 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b423d5fb-4d0b-4357-ad79-9e6877468e56" containerName="payload-processing" Oct 29 23:00:31.664321 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664311 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.664413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664387 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d701e362-cc92-46b6-a889-86b1d7caab3c" containerName="payload-pre-processing" Oct 29 23:00:31.664497 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664488 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:31.664567 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664559 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6efccd-b004-4083-a897-6454c274e4fb" containerName="payload-processing" Oct 29 23:00:31.664662 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664653 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2aaecce-4ed6-4eb6-83b1-0ab6667f1244" containerName="cleanup" Oct 29 23:00:31.664740 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.664731 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="37858cc2-c0c8-4d60-b15c-53f65745d14e" containerName="revoke-keys" Oct 29 23:00:31.671493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.671472 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.689677 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.689351 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2d286f-openshift-default-dockercfg-sjwfs\"" Oct 29 23:00:31.712352 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.712315 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l"] Oct 29 23:00:31.779151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.779112 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-socket\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.781118 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.781079 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e83482a-0401-47df-ab5a-1b419b3c371d-istiod-ca-cert\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.786453 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.781360 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-certs\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.786817 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.786792 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-token\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.786978 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.786962 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-podinfo\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.788564 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.787219 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6l5q\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-kube-api-access-x6l5q\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.788791 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.788770 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-credential-socket\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.788913 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.788895 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-data\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.789511 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.789483 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-envoy\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.821193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.821156 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902288 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-envoy\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902406 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-socket\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902470 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e83482a-0401-47df-ab5a-1b419b3c371d-istiod-ca-cert\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902533 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-certs\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902636 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-token\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902664 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-podinfo\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902780 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6l5q\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-kube-api-access-x6l5q\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902833 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-credential-socket\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.902865 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-data\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.905416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.903521 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-data\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.929466 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.908062 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-socket\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.929466 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.908807 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e83482a-0401-47df-ab5a-1b419b3c371d-istiod-ca-cert\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.929466 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.909052 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-certs\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.934431 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.931132 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-podinfo\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.938420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.934736 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-credential-socket\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.949486 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.949197 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-envoy\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.983450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.966145 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-token\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:31.983840 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:31.979147 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6l5q\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-kube-api-access-x6l5q\") pod \"e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:32.019957 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.019183 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:32.030419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.025170 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:32.225238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.225197 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:32.374860 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.374831 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l"] Oct 29 23:00:32.376787 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:00:32.376757 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e83482a_0401_47df_ab5a_1b419b3c371d.slice/crio-40f0d78f0c9c6c41da4f040f1aa65cef147e30e25509cdf596e3a8fa1cc9736b WatchSource:0}: Error finding container 40f0d78f0c9c6c41da4f040f1aa65cef147e30e25509cdf596e3a8fa1cc9736b: Status 404 returned error can't find the container with id 40f0d78f0c9c6c41da4f040f1aa65cef147e30e25509cdf596e3a8fa1cc9736b Oct 29 23:00:32.379930 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.379898 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:32.380042 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.379979 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:32.380042 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.380031 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:32.665470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.664892 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" event={"ID":"0e83482a-0401-47df-ab5a-1b419b3c371d","Type":"ContainerStarted","Data":"44d98ba7967fcd6dfc9c63a27f14cee4be9ee087e3153fdc061607c23f1f071e"} Oct 29 23:00:32.665470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.664966 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" event={"ID":"0e83482a-0401-47df-ab5a-1b419b3c371d","Type":"ContainerStarted","Data":"40f0d78f0c9c6c41da4f040f1aa65cef147e30e25509cdf596e3a8fa1cc9736b"} Oct 29 23:00:32.732846 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.732788 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podStartSLOduration=1.732771129 podStartE2EDuration="1.732771129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:32.730204991 +0000 UTC m=+2695.211522498" watchObservedRunningTime="2026-10-29 23:00:32.732771129 +0000 UTC m=+2695.214088621" Oct 29 23:00:32.736628 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:32.732954 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:33.019643 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:33.019608 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:33.022017 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:33.021985 5610 patch_prober.go:29] interesting pod/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 23:00:33.022168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:33.022045 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 23:00:33.883030 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:33.882897 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:34.024891 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:34.024858 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:34.684475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:34.684432 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:34.685443 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:34.685384 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:00:35.235043 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:35.235006 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:35.475931 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:35.475896 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:35.718702 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:35.718649 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" podUID="363eca02-06d0-4681-b9bb-057ecff62e64" containerName="payload-processing" containerID="cri-o://6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3" gracePeriod=25 Oct 29 23:00:35.754068 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:35.754031 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" podUID="eefa47b1-a8dd-40be-8d66-e411c9408d66" containerName="payload-pre-processing" containerID="cri-o://8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a" gracePeriod=25 Oct 29 23:00:36.005953 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.005834 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:36.219083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.218575 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 23:00:36.261035 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.260959 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 23:00:36.288233 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.288198 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefa47b1-a8dd-40be-8d66-e411c9408d66-kube-api-access-5gt7t\" (UniqueName: \"kubernetes.io/projected/eefa47b1-a8dd-40be-8d66-e411c9408d66-kube-api-access-5gt7t\") pod \"eefa47b1-a8dd-40be-8d66-e411c9408d66\" (UID: \"eefa47b1-a8dd-40be-8d66-e411c9408d66\") " Oct 29 23:00:36.288989 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.288272 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363eca02-06d0-4681-b9bb-057ecff62e64-kube-api-access-qg225\" (UniqueName: \"kubernetes.io/projected/363eca02-06d0-4681-b9bb-057ecff62e64-kube-api-access-qg225\") pod \"363eca02-06d0-4681-b9bb-057ecff62e64\" (UID: \"363eca02-06d0-4681-b9bb-057ecff62e64\") " Oct 29 23:00:36.288989 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.288320 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363eca02-06d0-4681-b9bb-057ecff62e64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363eca02-06d0-4681-b9bb-057ecff62e64-plugins-config-volume\") pod \"363eca02-06d0-4681-b9bb-057ecff62e64\" (UID: \"363eca02-06d0-4681-b9bb-057ecff62e64\") " Oct 29 23:00:36.288989 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.288575 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefa47b1-a8dd-40be-8d66-e411c9408d66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefa47b1-a8dd-40be-8d66-e411c9408d66-plugins-config-volume\") pod \"eefa47b1-a8dd-40be-8d66-e411c9408d66\" (UID: \"eefa47b1-a8dd-40be-8d66-e411c9408d66\") " Oct 29 23:00:36.289370 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.289345 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363eca02-06d0-4681-b9bb-057ecff62e64-plugins-config-volume" pod "363eca02-06d0-4681-b9bb-057ecff62e64" (UID: "363eca02-06d0-4681-b9bb-057ecff62e64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:36.289685 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.289659 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefa47b1-a8dd-40be-8d66-e411c9408d66-plugins-config-volume" pod "eefa47b1-a8dd-40be-8d66-e411c9408d66" (UID: "eefa47b1-a8dd-40be-8d66-e411c9408d66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:00:36.290106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.290085 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363eca02-06d0-4681-b9bb-057ecff62e64-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:36.290186 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.290108 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eefa47b1-a8dd-40be-8d66-e411c9408d66-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:36.292741 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.292716 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefa47b1-a8dd-40be-8d66-e411c9408d66-kube-api-access-5gt7t" pod "eefa47b1-a8dd-40be-8d66-e411c9408d66" (UID: "eefa47b1-a8dd-40be-8d66-e411c9408d66"). InnerVolumeSpecName "kube-api-access-5gt7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:36.293499 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.293474 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363eca02-06d0-4681-b9bb-057ecff62e64-kube-api-access-qg225" pod "363eca02-06d0-4681-b9bb-057ecff62e64" (UID: "363eca02-06d0-4681-b9bb-057ecff62e64"). InnerVolumeSpecName "kube-api-access-qg225". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:00:36.383362 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.383322 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:36.391472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.391443 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gt7t\" (UniqueName: \"kubernetes.io/projected/eefa47b1-a8dd-40be-8d66-e411c9408d66-kube-api-access-5gt7t\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:36.391472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.391470 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg225\" (UniqueName: \"kubernetes.io/projected/363eca02-06d0-4681-b9bb-057ecff62e64-kube-api-access-qg225\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:00:36.702866 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.702835 5610 generic.go:356] "Generic (PLEG): container finished" podID="363eca02-06d0-4681-b9bb-057ecff62e64" containerID="6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3" exitCode=0 Oct 29 23:00:36.703091 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.702897 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" Oct 29 23:00:36.703091 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.702918 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" event={"ID":"363eca02-06d0-4681-b9bb-057ecff62e64","Type":"ContainerDied","Data":"6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3"} Oct 29 23:00:36.703091 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.703022 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd" event={"ID":"363eca02-06d0-4681-b9bb-057ecff62e64","Type":"ContainerDied","Data":"53d0226873dfb0fc642ba690b501633566ee190d91a121c6cbba44fd9e81d597"} Oct 29 23:00:36.703091 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.703068 5610 scope.go:122] "RemoveContainer" containerID="6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3" Oct 29 23:00:36.706228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.706202 5610 generic.go:356] "Generic (PLEG): container finished" podID="eefa47b1-a8dd-40be-8d66-e411c9408d66" containerID="8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a" exitCode=0 Oct 29 23:00:36.706375 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.706255 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" Oct 29 23:00:36.706375 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.706314 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" event={"ID":"eefa47b1-a8dd-40be-8d66-e411c9408d66","Type":"ContainerDied","Data":"8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a"} Oct 29 23:00:36.706506 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.706385 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8" event={"ID":"eefa47b1-a8dd-40be-8d66-e411c9408d66","Type":"ContainerDied","Data":"cdab8fef9ae4dd382f16669cb8bcc951b1bd43809f94f14918e8a3707eede991"} Oct 29 23:00:36.715917 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.715898 5610 scope.go:122] "RemoveContainer" containerID="6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3" Oct 29 23:00:36.716215 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:36.716188 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3\": container with ID starting with 6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3 not found: ID does not exist" containerID="6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3" Oct 29 23:00:36.716284 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.716226 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3"} err="failed to get container status \"6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3\": rpc error: code = NotFound desc = could not find container \"6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3\": container with ID starting with 6e224f1a8d11360f3780b3ebb6462d212b52dedefaeea77afe111d2f8d2298a3 not found: ID does not exist" Oct 29 23:00:36.716284 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.716249 5610 scope.go:122] "RemoveContainer" containerID="8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a" Oct 29 23:00:36.726644 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.726450 5610 scope.go:122] "RemoveContainer" containerID="8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a" Oct 29 23:00:36.726751 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:36.726731 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a\": container with ID starting with 8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a not found: ID does not exist" containerID="8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a" Oct 29 23:00:36.726793 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.726759 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a"} err="failed to get container status \"8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a\": rpc error: code = NotFound desc = could not find container \"8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a\": container with ID starting with 8ceadbb78636b4ea25ec61c4dbbc1f2c5d27d423a7f547486197d370e0fe003a not found: ID does not exist" Oct 29 23:00:36.769776 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.769738 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd"] Oct 29 23:00:36.778342 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.778314 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-59db99bc46-x2jcd"] Oct 29 23:00:36.808955 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.808881 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8"] Oct 29 23:00:36.812933 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:36.812905 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-14b0c3-6568595bb7-zjwr8"] Oct 29 23:00:37.637583 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:37.637551 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363eca02-06d0-4681-b9bb-057ecff62e64" path="/var/lib/kubelet/pods/363eca02-06d0-4681-b9bb-057ecff62e64/volumes" Oct 29 23:00:37.638042 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:37.638027 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefa47b1-a8dd-40be-8d66-e411c9408d66" path="/var/lib/kubelet/pods/eefa47b1-a8dd-40be-8d66-e411c9408d66/volumes" Oct 29 23:00:38.154690 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.154654 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:38.259695 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.259655 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6"] Oct 29 23:00:38.261604 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261575 5610 cpu_manager.go:414] "containerMap: removing container" podUID="eefa47b1-a8dd-40be-8d66-e411c9408d66" containerName="payload-pre-processing" Oct 29 23:00:38.261604 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261602 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefa47b1-a8dd-40be-8d66-e411c9408d66" containerName="payload-pre-processing" Oct 29 23:00:38.261816 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261622 5610 cpu_manager.go:414] "containerMap: removing container" podUID="363eca02-06d0-4681-b9bb-057ecff62e64" containerName="payload-processing" Oct 29 23:00:38.261816 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261630 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363eca02-06d0-4681-b9bb-057ecff62e64" containerName="payload-processing" Oct 29 23:00:38.261816 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261693 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:38.261816 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261701 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:38.262014 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261912 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce705b5-e86e-4ca9-919a-ff454c193ce9" containerName="cleanup" Oct 29 23:00:38.262014 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261934 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefa47b1-a8dd-40be-8d66-e411c9408d66" containerName="payload-pre-processing" Oct 29 23:00:38.262014 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.261946 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="363eca02-06d0-4681-b9bb-057ecff62e64" containerName="payload-processing" Oct 29 23:00:38.266862 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.266835 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:38.282721 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.282685 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d286f-dockercfg-vx92p\"" Oct 29 23:00:38.294684 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.294658 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d286f\"" Oct 29 23:00:38.299368 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.299341 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6"] Oct 29 23:00:38.325450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.325384 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/442517be-d8d2-4a32-b0c4-255bb0934adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6\" (UID: \"442517be-d8d2-4a32-b0c4-255bb0934adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:38.325765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.325741 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5b5\" (UniqueName: \"kubernetes.io/projected/442517be-d8d2-4a32-b0c4-255bb0934adc-kube-api-access-4r5b5\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6\" (UID: \"442517be-d8d2-4a32-b0c4-255bb0934adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:38.326765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.326734 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:38.531425 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.502925 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5b5\" (UniqueName: \"kubernetes.io/projected/442517be-d8d2-4a32-b0c4-255bb0934adc-kube-api-access-4r5b5\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6\" (UID: \"442517be-d8d2-4a32-b0c4-255bb0934adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:38.556446 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.532140 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/442517be-d8d2-4a32-b0c4-255bb0934adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6\" (UID: \"442517be-d8d2-4a32-b0c4-255bb0934adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:38.556943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.540826 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/442517be-d8d2-4a32-b0c4-255bb0934adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6\" (UID: \"442517be-d8d2-4a32-b0c4-255bb0934adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:38.614902 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.614870 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph"] Oct 29 23:00:38.618771 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.618742 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5b5\" (UniqueName: \"kubernetes.io/projected/442517be-d8d2-4a32-b0c4-255bb0934adc-kube-api-access-4r5b5\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6\" (UID: \"442517be-d8d2-4a32-b0c4-255bb0934adc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:38.630174 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.630128 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph"] Oct 29 23:00:38.630345 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.630277 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:38.661872 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.661836 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:38.663520 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.662602 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aead210-f299-4300-9978-f24becb36fed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph\" (UID: \"9aead210-f299-4300-9978-f24becb36fed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:38.663520 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.662740 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9857\" (UniqueName: \"kubernetes.io/projected/9aead210-f299-4300-9978-f24becb36fed-kube-api-access-q9857\") pod \"payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph\" (UID: \"9aead210-f299-4300-9978-f24becb36fed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:38.764023 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.763984 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aead210-f299-4300-9978-f24becb36fed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph\" (UID: \"9aead210-f299-4300-9978-f24becb36fed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:38.764203 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.764093 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9857\" (UniqueName: \"kubernetes.io/projected/9aead210-f299-4300-9978-f24becb36fed-kube-api-access-q9857\") pod \"payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph\" (UID: \"9aead210-f299-4300-9978-f24becb36fed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:38.764788 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.764765 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aead210-f299-4300-9978-f24becb36fed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph\" (UID: \"9aead210-f299-4300-9978-f24becb36fed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:38.782313 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.782212 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9857\" (UniqueName: \"kubernetes.io/projected/9aead210-f299-4300-9978-f24becb36fed-kube-api-access-q9857\") pod \"payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph\" (UID: \"9aead210-f299-4300-9978-f24becb36fed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:38.889232 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.889194 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:38.954169 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:38.954135 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:39.223432 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.223379 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6"] Oct 29 23:00:39.223554 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:00:39.223498 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442517be_d8d2_4a32_b0c4_255bb0934adc.slice/crio-e0d233d14b1f19ac00fd84b1bd18dad85539a1bf513675f886dbb8a630837f20 WatchSource:0}: Error finding container e0d233d14b1f19ac00fd84b1bd18dad85539a1bf513675f886dbb8a630837f20: Status 404 returned error can't find the container with id e0d233d14b1f19ac00fd84b1bd18dad85539a1bf513675f886dbb8a630837f20 Oct 29 23:00:39.296629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.296585 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:39.402413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.399101 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph"] Oct 29 23:00:39.749623 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.749575 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" event={"ID":"9aead210-f299-4300-9978-f24becb36fed","Type":"ContainerStarted","Data":"a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8"} Oct 29 23:00:39.750106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.749638 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" event={"ID":"9aead210-f299-4300-9978-f24becb36fed","Type":"ContainerStarted","Data":"29fb8577addfdcc42e636dc01e8644909e02ebb04dad4e90c1bcda6bc75e7a19"} Oct 29 23:00:39.750106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.749680 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:39.751432 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.751376 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" event={"ID":"442517be-d8d2-4a32-b0c4-255bb0934adc","Type":"ContainerStarted","Data":"ba9c67c795f8115c3a53f33508e57815b5b694df6843ad895d657ab633a7f86c"} Oct 29 23:00:39.751569 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.751452 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" event={"ID":"442517be-d8d2-4a32-b0c4-255bb0934adc","Type":"ContainerStarted","Data":"e0d233d14b1f19ac00fd84b1bd18dad85539a1bf513675f886dbb8a630837f20"} Oct 29 23:00:39.751569 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.751492 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:39.794223 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.794065 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" podStartSLOduration=1.7940454030000001 podStartE2EDuration="1.794045403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:39.789429413 +0000 UTC m=+2702.270746910" watchObservedRunningTime="2026-10-29 23:00:39.794045403 +0000 UTC m=+2702.275362897" Oct 29 23:00:39.830091 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.830035 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" podStartSLOduration=1.830023307 podStartE2EDuration="1.830023307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:39.827730356 +0000 UTC m=+2702.309047850" watchObservedRunningTime="2026-10-29 23:00:39.830023307 +0000 UTC m=+2702.311340799" Oct 29 23:00:39.870189 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:39.870156 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:40.221111 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.221077 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw"] Oct 29 23:00:40.226711 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.226673 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.242540 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.242477 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9ee1fc-openshift-default-dockercfg-8668w\"" Oct 29 23:00:40.257182 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.257082 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw"] Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.374534 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-envoy\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.374607 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-data\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.374650 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-credential-socket\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.374844 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-socket\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.374900 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-istio-token\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.374981 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77602a7-cf53-413f-9484-7baccce25ea5-istiod-ca-cert\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.375040 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77602a7-cf53-413f-9484-7baccce25ea5-istio-podinfo\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.375077 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sztjc\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-kube-api-access-sztjc\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.378417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.375152 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-certs\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.446430 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.440719 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492454 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-data\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492517 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-credential-socket\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492645 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-socket\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492686 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-istio-token\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492732 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77602a7-cf53-413f-9484-7baccce25ea5-istiod-ca-cert\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492769 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77602a7-cf53-413f-9484-7baccce25ea5-istio-podinfo\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492801 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sztjc\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-kube-api-access-sztjc\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492854 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-certs\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.493475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.492970 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-envoy\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.498422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.494503 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-data\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.498422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.494839 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-credential-socket\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.498422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.495085 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-socket\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.512547 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.500121 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77602a7-cf53-413f-9484-7baccce25ea5-istiod-ca-cert\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.512547 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.500709 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-certs\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.512547 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.502978 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-envoy\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.553318 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.546044 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77602a7-cf53-413f-9484-7baccce25ea5-istio-podinfo\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.564322 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.564285 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-istio-token\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.568999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.568965 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sztjc\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-kube-api-access-sztjc\") pod \"e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.719116 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.719083 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:40.854366 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.853692 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:40.903229 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:40.903191 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:41.993449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:41.991129 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw"] Oct 29 23:00:42.089422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:42.086743 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:42.089422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:42.086837 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:42.089422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:42.086878 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:42.762416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:42.757371 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:00:43.124126 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:43.122464 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" event={"ID":"e77602a7-cf53-413f-9484-7baccce25ea5","Type":"ContainerStarted","Data":"d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa"} Oct 29 23:00:43.124126 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:43.122540 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" event={"ID":"e77602a7-cf53-413f-9484-7baccce25ea5","Type":"ContainerStarted","Data":"108cf3d74744bfe22280ae3fa7f94e40525123763fd0676c9e83a90d3d8f48f4"} Oct 29 23:00:43.127071 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:43.127013 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:43.209863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:43.209728 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" podStartSLOduration=3.209709752 podStartE2EDuration="3.209709752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:43.189419812 +0000 UTC m=+2705.670737308" watchObservedRunningTime="2026-10-29 23:00:43.209709752 +0000 UTC m=+2705.691027242" Oct 29 23:00:43.714845 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:00:43.714806 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:00:43.855540 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:43.855508 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:43.861555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:43.861527 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:44.142877 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:44.142781 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:44.143875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:44.143855 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:00:45.075417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:45.067150 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:45.640996 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:45.640958 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:45.817147 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:45.817110 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:47.124179 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:47.124136 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:47.601084 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:47.601048 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:47.826421 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:47.826150 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:49.025984 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:49.025945 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:50.687474 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:50.687025 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:50.765084 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:50.765054 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:00:50.765284 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:50.765130 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:00:52.435039 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:52.435001 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:53.494491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:53.494450 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9"] Oct 29 23:00:53.595727 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:53.595692 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:53.852693 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:53.852613 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m"] Oct 29 23:00:53.863045 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:53.861714 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m"] Oct 29 23:00:53.863045 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:53.861832 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:53.877970 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:53.877873 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-40cf9a-openshift-default-dockercfg-8bd6l\"" Oct 29 23:00:54.012337 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012304 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.012546 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012354 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgb74\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-kube-api-access-pgb74\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.012546 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012519 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.012659 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012550 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.012659 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012579 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.012762 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012739 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.012813 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012789 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.012903 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012885 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.012971 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.012948 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.118046 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.117937 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.135567 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.135523 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.136923 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.136895 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.141791 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.141760 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.142209 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.142175 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.147546 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.147505 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgb74\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-kube-api-access-pgb74\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.148269 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.148245 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.179857 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.149582 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.180236 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.143576 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.180849 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.180824 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.181025 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.119055 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istiod-ca-cert\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.181152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.149958 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-certs\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.181281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.158504 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-envoy\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.181432 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.148724 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-credential-socket\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.181580 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.180192 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-data\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.195080 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.195052 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-podinfo\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.206458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.204440 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:54.206458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.204918 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-token\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.206458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.205501 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgb74\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-kube-api-access-pgb74\") pod \"e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.487199 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.487164 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:54.489196 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.489162 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:54.943003 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.942969 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m"] Oct 29 23:00:54.944072 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:00:54.944043 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219cb1bb_3ba5_4b78_b457_0f4793cb4acf.slice/crio-7b27eeb18be82a5afdca6fbe154e4bb50e62fe93751f7321c2789640e1981c14 WatchSource:0}: Error finding container 7b27eeb18be82a5afdca6fbe154e4bb50e62fe93751f7321c2789640e1981c14: Status 404 returned error can't find the container with id 7b27eeb18be82a5afdca6fbe154e4bb50e62fe93751f7321c2789640e1981c14 Oct 29 23:00:54.947067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.947034 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:54.947153 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.947099 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:54.947153 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:54.947130 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:00:55.330832 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:55.330794 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" event={"ID":"219cb1bb-3ba5-4b78-b457-0f4793cb4acf","Type":"ContainerStarted","Data":"e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784"} Oct 29 23:00:55.331038 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:55.330861 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" event={"ID":"219cb1bb-3ba5-4b78-b457-0f4793cb4acf","Type":"ContainerStarted","Data":"7b27eeb18be82a5afdca6fbe154e4bb50e62fe93751f7321c2789640e1981c14"} Oct 29 23:00:55.454573 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:55.454495 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" podStartSLOduration=2.454474706 podStartE2EDuration="2.454474706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:55.380934924 +0000 UTC m=+2717.862252417" watchObservedRunningTime="2026-10-29 23:00:55.454474706 +0000 UTC m=+2717.935792201" Oct 29 23:00:55.460441 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:55.460271 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:55.488695 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:55.488665 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:00:55.806335 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:55.806300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:56.494017 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:56.493966 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:00:56.494529 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:56.494031 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:00:57.108418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:57.104969 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:57.496366 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:57.495569 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:00:57.496366 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:57.495630 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:00:58.488953 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:58.488912 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:00:58.489178 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:58.488966 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:00:58.824764 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:58.824686 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:59.126429 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:59.126321 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:59.351974 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:59.351941 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:00:59.489247 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:59.489207 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:00:59.489448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:00:59.489270 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:01:00.498547 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:00.498510 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:01:00.499165 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:00.499134 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:01:00.821467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:00.819002 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:01.037079 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:01.036967 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:01.489580 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:01.489535 5610 patch_prober.go:29] interesting pod/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 29 23:01:01.489808 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:01.489599 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 29 23:01:01.914084 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:01.913995 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:02.494634 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:02.494606 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:01:02.495150 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:02.495114 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:01:02.495979 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:02.495959 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:01:03.585419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:03.584253 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:03.909434 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:03.909325 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm"] Oct 29 23:01:03.916347 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:03.916318 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:03.955950 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:03.955918 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0906f4-dockercfg-z6wsw\"" Oct 29 23:01:03.957865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:03.956739 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0906f4\"" Oct 29 23:01:03.985069 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:03.976839 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d156d3-b215-4564-9052-db75adf96095-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm\" (UID: \"43d156d3-b215-4564-9052-db75adf96095\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:03.985069 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:03.976954 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj45g\" (UniqueName: \"kubernetes.io/projected/43d156d3-b215-4564-9052-db75adf96095-kube-api-access-fj45g\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm\" (UID: \"43d156d3-b215-4564-9052-db75adf96095\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:03.995018 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:03.994943 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm"] Oct 29 23:01:04.078380 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.078333 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d156d3-b215-4564-9052-db75adf96095-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm\" (UID: \"43d156d3-b215-4564-9052-db75adf96095\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:04.079867 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.079802 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj45g\" (UniqueName: \"kubernetes.io/projected/43d156d3-b215-4564-9052-db75adf96095-kube-api-access-fj45g\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm\" (UID: \"43d156d3-b215-4564-9052-db75adf96095\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:04.080353 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.079679 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d156d3-b215-4564-9052-db75adf96095-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm\" (UID: \"43d156d3-b215-4564-9052-db75adf96095\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:04.084381 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.084352 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f"] Oct 29 23:01:04.090038 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.090021 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:04.117595 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.117569 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj45g\" (UniqueName: \"kubernetes.io/projected/43d156d3-b215-4564-9052-db75adf96095-kube-api-access-fj45g\") pod \"payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm\" (UID: \"43d156d3-b215-4564-9052-db75adf96095\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:04.126865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.126835 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f"] Oct 29 23:01:04.182439 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.182332 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct5ph\" (UniqueName: \"kubernetes.io/projected/91a0bb3e-f76d-4030-a53c-ac5905587c93-kube-api-access-ct5ph\") pod \"payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f\" (UID: \"91a0bb3e-f76d-4030-a53c-ac5905587c93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:04.182636 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.182489 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a0bb3e-f76d-4030-a53c-ac5905587c93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f\" (UID: \"91a0bb3e-f76d-4030-a53c-ac5905587c93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:04.274793 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.274757 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:04.284327 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.284299 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a0bb3e-f76d-4030-a53c-ac5905587c93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f\" (UID: \"91a0bb3e-f76d-4030-a53c-ac5905587c93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:04.284552 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.284536 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct5ph\" (UniqueName: \"kubernetes.io/projected/91a0bb3e-f76d-4030-a53c-ac5905587c93-kube-api-access-ct5ph\") pod \"payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f\" (UID: \"91a0bb3e-f76d-4030-a53c-ac5905587c93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:04.285217 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.285195 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a0bb3e-f76d-4030-a53c-ac5905587c93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f\" (UID: \"91a0bb3e-f76d-4030-a53c-ac5905587c93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:04.305062 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.305028 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct5ph\" (UniqueName: \"kubernetes.io/projected/91a0bb3e-f76d-4030-a53c-ac5905587c93-kube-api-access-ct5ph\") pod \"payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f\" (UID: \"91a0bb3e-f76d-4030-a53c-ac5905587c93\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:04.314247 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.314222 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:04.407385 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.407349 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:04.597612 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.597583 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:04.637828 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.637806 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm"] Oct 29 23:01:04.824690 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:01:04.824654 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a0bb3e_f76d_4030_a53c_ac5905587c93.slice/crio-0aba4fb966df747c3d2e24e7c0d999cb3f6638e3ec5678cc0fd2c01895c8c5bf WatchSource:0}: Error finding container 0aba4fb966df747c3d2e24e7c0d999cb3f6638e3ec5678cc0fd2c01895c8c5bf: Status 404 returned error can't find the container with id 0aba4fb966df747c3d2e24e7c0d999cb3f6638e3ec5678cc0fd2c01895c8c5bf Oct 29 23:01:04.826909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:04.826862 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f"] Oct 29 23:01:05.334488 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.334378 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p"] Oct 29 23:01:05.350743 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.350709 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:05.409012 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.408948 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n"] Oct 29 23:01:05.613240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.613147 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" event={"ID":"91a0bb3e-f76d-4030-a53c-ac5905587c93","Type":"ContainerStarted","Data":"83716364df0d8911b542caa7e49ee33d89b6032fdd853d8ddc5898af0cef21cf"} Oct 29 23:01:05.613875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.613849 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" event={"ID":"91a0bb3e-f76d-4030-a53c-ac5905587c93","Type":"ContainerStarted","Data":"0aba4fb966df747c3d2e24e7c0d999cb3f6638e3ec5678cc0fd2c01895c8c5bf"} Oct 29 23:01:05.614559 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.614540 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:05.617300 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.617276 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" event={"ID":"43d156d3-b215-4564-9052-db75adf96095","Type":"ContainerStarted","Data":"8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d"} Oct 29 23:01:05.617482 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.617467 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" event={"ID":"43d156d3-b215-4564-9052-db75adf96095","Type":"ContainerStarted","Data":"4a88dfb44233fc3e8464416dfb96fd4539c93d655ecd337a767c04dc925d7ad0"} Oct 29 23:01:05.618207 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.618188 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:05.769172 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.769109 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" podStartSLOduration=1.769090093 podStartE2EDuration="1.769090093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:05.767660164 +0000 UTC m=+2728.248977658" watchObservedRunningTime="2026-10-29 23:01:05.769090093 +0000 UTC m=+2728.250407589" Oct 29 23:01:05.881266 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:05.881152 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" podStartSLOduration=2.881132659 podStartE2EDuration="2.881132659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:05.877059987 +0000 UTC m=+2728.358377480" watchObservedRunningTime="2026-10-29 23:01:05.881132659 +0000 UTC m=+2728.362450153" Oct 29 23:01:06.013471 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:06.012765 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:06.540417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:06.521011 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:08.256732 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.256697 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:08.516733 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.516641 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" podUID="f97607e1-3c37-4b30-99ef-e9d2fe142938" containerName="main" containerID="cri-o://c04cec3636736c3d04d00248186da738e163dde88c373f749af36f51b5dcc24b" gracePeriod=45 Oct 29 23:01:08.563356 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.563316 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" podUID="f97607e1-3c37-4b30-99ef-e9d2fe142938" containerName="main" probeResult="failure" output="Get \"https://10.134.0.219:8000/ready\": dial tcp 10.134.0.219:8000: connect: connection refused" Oct 29 23:01:08.648376 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.648348 5610 generic.go:356] "Generic (PLEG): container finished" podID="f97607e1-3c37-4b30-99ef-e9d2fe142938" containerID="c04cec3636736c3d04d00248186da738e163dde88c373f749af36f51b5dcc24b" exitCode=0 Oct 29 23:01:08.648550 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.648430 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" event={"ID":"f97607e1-3c37-4b30-99ef-e9d2fe142938","Type":"ContainerDied","Data":"c04cec3636736c3d04d00248186da738e163dde88c373f749af36f51b5dcc24b"} Oct 29 23:01:08.805297 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.805269 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 23:01:08.855433 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.855386 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-dshm\") pod \"f97607e1-3c37-4b30-99ef-e9d2fe142938\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " Oct 29 23:01:08.855599 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.855486 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-model-cache\") pod \"f97607e1-3c37-4b30-99ef-e9d2fe142938\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " Oct 29 23:01:08.855685 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.855599 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f97607e1-3c37-4b30-99ef-e9d2fe142938-tls-certs\" (UniqueName: \"kubernetes.io/secret/f97607e1-3c37-4b30-99ef-e9d2fe142938-tls-certs\") pod \"f97607e1-3c37-4b30-99ef-e9d2fe142938\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " Oct 29 23:01:08.855768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.855700 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-home\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-home\") pod \"f97607e1-3c37-4b30-99ef-e9d2fe142938\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " Oct 29 23:01:08.855831 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.855772 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-tmp-dir\") pod \"f97607e1-3c37-4b30-99ef-e9d2fe142938\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " Oct 29 23:01:08.855831 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.855782 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-model-cache" pod "f97607e1-3c37-4b30-99ef-e9d2fe142938" (UID: "f97607e1-3c37-4b30-99ef-e9d2fe142938"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:08.856014 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.855926 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97607e1-3c37-4b30-99ef-e9d2fe142938-kube-api-access-9wf4t\" (UniqueName: \"kubernetes.io/projected/f97607e1-3c37-4b30-99ef-e9d2fe142938-kube-api-access-9wf4t\") pod \"f97607e1-3c37-4b30-99ef-e9d2fe142938\" (UID: \"f97607e1-3c37-4b30-99ef-e9d2fe142938\") " Oct 29 23:01:08.856014 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.855998 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-home" pod "f97607e1-3c37-4b30-99ef-e9d2fe142938" (UID: "f97607e1-3c37-4b30-99ef-e9d2fe142938"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:08.856192 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.856166 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-tmp-dir" pod "f97607e1-3c37-4b30-99ef-e9d2fe142938" (UID: "f97607e1-3c37-4b30-99ef-e9d2fe142938"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:08.856696 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.856667 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:08.856823 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.856696 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:08.856823 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.856711 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:08.859807 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.859779 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97607e1-3c37-4b30-99ef-e9d2fe142938-tls-certs" pod "f97607e1-3c37-4b30-99ef-e9d2fe142938" (UID: "f97607e1-3c37-4b30-99ef-e9d2fe142938"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:08.859807 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.859783 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-dshm" pod "f97607e1-3c37-4b30-99ef-e9d2fe142938" (UID: "f97607e1-3c37-4b30-99ef-e9d2fe142938"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:08.860638 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.860618 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97607e1-3c37-4b30-99ef-e9d2fe142938-kube-api-access-9wf4t" pod "f97607e1-3c37-4b30-99ef-e9d2fe142938" (UID: "f97607e1-3c37-4b30-99ef-e9d2fe142938"). InnerVolumeSpecName "kube-api-access-9wf4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:08.958470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.958435 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97607e1-3c37-4b30-99ef-e9d2fe142938-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:08.958470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.958467 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wf4t\" (UniqueName: \"kubernetes.io/projected/f97607e1-3c37-4b30-99ef-e9d2fe142938-kube-api-access-9wf4t\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:08.958470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:08.958478 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97607e1-3c37-4b30-99ef-e9d2fe142938-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:09.658506 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:09.658467 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" Oct 29 23:01:09.658506 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:09.658484 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9" event={"ID":"f97607e1-3c37-4b30-99ef-e9d2fe142938","Type":"ContainerDied","Data":"5e888d3acbf9145c1b2d22546787ead5f79d27bdb9326327c3aa3a7389a494d4"} Oct 29 23:01:09.659032 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:09.658550 5610 scope.go:122] "RemoveContainer" containerID="c04cec3636736c3d04d00248186da738e163dde88c373f749af36f51b5dcc24b" Oct 29 23:01:09.731302 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:09.731271 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9"] Oct 29 23:01:09.743577 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:09.743540 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8f1627/auth-test-model-8f1627-kserve-6ddb4675b5-smhp9"] Oct 29 23:01:10.335452 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:10.335339 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" podUID="a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" containerName="payload-processing" containerID="cri-o://06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378" gracePeriod=25 Oct 29 23:01:10.409982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:10.409938 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" podUID="8566790e-0736-4bbf-845f-3df7bbb82853" containerName="payload-pre-processing" containerID="cri-o://cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d" gracePeriod=25 Oct 29 23:01:10.413477 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:10.413443 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:11.641726 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:11.641691 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97607e1-3c37-4b30-99ef-e9d2fe142938" path="/var/lib/kubelet/pods/f97607e1-3c37-4b30-99ef-e9d2fe142938/volumes" Oct 29 23:01:12.135683 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:12.135647 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:12.608829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:12.608778 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" podUID="a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:12.610183 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:12.610142 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" podUID="8566790e-0736-4bbf-845f-3df7bbb82853" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:12.831761 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:12.831724 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:13.472763 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:13.472727 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:13.719076 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:13.719027 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:01:13.781778 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:13.781660 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:15.307592 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.307555 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q"] Oct 29 23:01:15.308631 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.308606 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f97607e1-3c37-4b30-99ef-e9d2fe142938" containerName="main" Oct 29 23:01:15.308631 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.308630 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97607e1-3c37-4b30-99ef-e9d2fe142938" containerName="main" Oct 29 23:01:15.308855 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.308840 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97607e1-3c37-4b30-99ef-e9d2fe142938" containerName="main" Oct 29 23:01:15.314365 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.314341 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.358555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.358520 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q"] Oct 29 23:01:15.473854 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.473814 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.474072 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.473948 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.474072 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.474009 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkvkc\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-kube-api-access-hkvkc\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.474072 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.474044 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.474246 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.474208 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.474334 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.474308 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.474499 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.474483 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.474620 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.474598 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.474668 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.474652 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576109 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576019 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576109 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576062 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576351 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576131 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576351 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576185 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkvkc\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-kube-api-access-hkvkc\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576351 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576222 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576351 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576285 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576351 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576345 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576510 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576518 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-credential-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576619 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-certs\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576631 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.576916 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.576705 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-data\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.577034 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.577009 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-socket\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.577308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.577282 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istiod-ca-cert\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.578834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.578803 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-envoy\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.579453 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.579432 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-podinfo\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.609652 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.609628 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-token\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.609769 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.609748 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkvkc\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-kube-api-access-hkvkc\") pod \"e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.630877 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.630850 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:15.921665 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.921572 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:15.997643 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:15.997598 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q"] Oct 29 23:01:16.003935 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:01:16.003893 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d34d23d_38d1_4525_bcc1_2a84d5c32724.slice/crio-b9223dd1f1c05bf55845474fd73f92fef7ee84881183082eba37591f666705cb WatchSource:0}: Error finding container b9223dd1f1c05bf55845474fd73f92fef7ee84881183082eba37591f666705cb: Status 404 returned error can't find the container with id b9223dd1f1c05bf55845474fd73f92fef7ee84881183082eba37591f666705cb Oct 29 23:01:16.011049 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:16.011015 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:16.011159 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:16.011104 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:16.011159 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:16.011153 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:01:16.629560 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:16.629531 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:01:16.732160 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:16.732124 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" event={"ID":"5d34d23d-38d1-4525-bcc1-2a84d5c32724","Type":"ContainerStarted","Data":"8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8"} Oct 29 23:01:16.732339 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:16.732177 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" event={"ID":"5d34d23d-38d1-4525-bcc1-2a84d5c32724","Type":"ContainerStarted","Data":"b9223dd1f1c05bf55845474fd73f92fef7ee84881183082eba37591f666705cb"} Oct 29 23:01:16.790418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:16.790348 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" podStartSLOduration=1.79033427 podStartE2EDuration="1.79033427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:16.787374056 +0000 UTC m=+2739.268691550" watchObservedRunningTime="2026-10-29 23:01:16.79033427 +0000 UTC m=+2739.271651763" Oct 29 23:01:17.638969 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:17.638919 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:01:17.639445 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:17.638999 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 29 23:01:17.644652 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:17.644630 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:01:17.644789 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:17.644673 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:17.850058 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:17.849989 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:18.046523 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:18.046485 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m"] Oct 29 23:01:18.047212 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:18.047170 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" containerID="cri-o://e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784" gracePeriod=30 Oct 29 23:01:18.219973 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:18.219880 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:18.631186 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:18.631156 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:01:18.631368 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:18.631212 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 29 23:01:19.631304 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:19.631267 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:01:19.631779 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:19.631316 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 29 23:01:19.966417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:19.966164 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:20.228689 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.228612 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:20.534542 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.534464 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc"] Oct 29 23:01:20.539760 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.539740 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:20.556313 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.556274 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-14b0c3\"" Oct 29 23:01:20.599852 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.599817 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-14b0c3-dockercfg-w77ts\"" Oct 29 23:01:20.624947 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.624915 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc"] Oct 29 23:01:20.631882 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.631848 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:01:20.632554 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.632523 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 29 23:01:20.747022 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.746990 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdgzb\" (UniqueName: \"kubernetes.io/projected/7e826947-7906-41b9-9bd5-2147dbb0c076-kube-api-access-qdgzb\") pod \"payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc\" (UID: \"7e826947-7906-41b9-9bd5-2147dbb0c076\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:20.747358 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.747341 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e826947-7906-41b9-9bd5-2147dbb0c076-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc\" (UID: \"7e826947-7906-41b9-9bd5-2147dbb0c076\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:20.852830 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.851213 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e826947-7906-41b9-9bd5-2147dbb0c076-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc\" (UID: \"7e826947-7906-41b9-9bd5-2147dbb0c076\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:20.854719 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.853064 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdgzb\" (UniqueName: \"kubernetes.io/projected/7e826947-7906-41b9-9bd5-2147dbb0c076-kube-api-access-qdgzb\") pod \"payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc\" (UID: \"7e826947-7906-41b9-9bd5-2147dbb0c076\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:20.854719 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.854666 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e826947-7906-41b9-9bd5-2147dbb0c076-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc\" (UID: \"7e826947-7906-41b9-9bd5-2147dbb0c076\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:20.947821 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:20.947755 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdgzb\" (UniqueName: \"kubernetes.io/projected/7e826947-7906-41b9-9bd5-2147dbb0c076-kube-api-access-qdgzb\") pod \"payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc\" (UID: \"7e826947-7906-41b9-9bd5-2147dbb0c076\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:21.203149 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.203110 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:21.601450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.600415 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc"] Oct 29 23:01:21.602494 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:01:21.602459 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e826947_7906_41b9_9bd5_2147dbb0c076.slice/crio-c10ef1da355f3f56d6d6f27924ac7974db9ac2473517a616898d8fc5c1608fbe WatchSource:0}: Error finding container c10ef1da355f3f56d6d6f27924ac7974db9ac2473517a616898d8fc5c1608fbe: Status 404 returned error can't find the container with id c10ef1da355f3f56d6d6f27924ac7974db9ac2473517a616898d8fc5c1608fbe Oct 29 23:01:21.631708 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.631668 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:01:21.631827 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.631723 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 29 23:01:21.840353 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.840319 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" event={"ID":"7e826947-7906-41b9-9bd5-2147dbb0c076","Type":"ContainerStarted","Data":"4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2"} Oct 29 23:01:21.840848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.840374 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" event={"ID":"7e826947-7906-41b9-9bd5-2147dbb0c076","Type":"ContainerStarted","Data":"c10ef1da355f3f56d6d6f27924ac7974db9ac2473517a616898d8fc5c1608fbe"} Oct 29 23:01:21.840848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.840605 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:21.910005 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.909865 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:21.941239 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:21.941086 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" podStartSLOduration=1.941065839 podStartE2EDuration="1.941065839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:21.908682139 +0000 UTC m=+2744.389999634" watchObservedRunningTime="2026-10-29 23:01:21.941065839 +0000 UTC m=+2744.422383334" Oct 29 23:01:22.604165 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.604098 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" podUID="a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:22.604165 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.604135 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" podUID="8566790e-0736-4bbf-845f-3df7bbb82853" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:22.631642 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.631606 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 29 23:01:22.631802 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.631671 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 29 23:01:22.670009 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.669967 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:22.884733 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.884630 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:22.958943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.958899 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4"] Oct 29 23:01:22.959184 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.959157 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" containerID="cri-o://0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb" gracePeriod=30 Oct 29 23:01:22.985435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.984708 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q"] Oct 29 23:01:22.985435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:22.985043 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" containerID="cri-o://8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8" gracePeriod=30 Oct 29 23:01:23.103261 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.098145 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:23.537661 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.537615 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl"] Oct 29 23:01:23.538702 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.538669 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" podUID="8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" containerName="maas-api" containerID="cri-o://c75d860b197105167fd2268f6657785086f4217b98146fe92871568d3d1930c5" gracePeriod=30 Oct 29 23:01:23.684881 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.682022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:23.684881 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.684141 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.815922 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-podinfo\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.815964 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-socket\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.816883 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-socket" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.817124 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-credential-socket\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.817171 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-kube-api-access-pgb74\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-kube-api-access-pgb74\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.817215 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-token\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-token\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.817639 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n"] Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.817714 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-credential-socket" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.817857 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" podUID="8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee" containerName="payload-pre-processing" containerID="cri-o://bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11" gracePeriod=30 Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.818755 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-certs\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.818922 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istiod-ca-cert\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.818974 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-data\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.819001 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-envoy\") pod \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\" (UID: \"219cb1bb-3ba5-4b78-b457-0f4793cb4acf\") " Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.820170 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istiod-ca-cert" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.820405 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-certs" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.822768 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.822792 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.822807 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.822821 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.823226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.823049 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-data" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.828340 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.828316 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-podinfo" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:23.828711 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.828690 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-token" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:23.835480 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.833780 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-envoy" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:23.859763 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.858661 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-kube-api-access-pgb74" pod "219cb1bb-3ba5-4b78-b457-0f4793cb4acf" (UID: "219cb1bb-3ba5-4b78-b457-0f4793cb4acf"). InnerVolumeSpecName "kube-api-access-pgb74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:23.883826 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.880442 5610 generic.go:356] "Generic (PLEG): container finished" podID="8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" containerID="c75d860b197105167fd2268f6657785086f4217b98146fe92871568d3d1930c5" exitCode=0 Oct 29 23:01:23.883826 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.880760 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" event={"ID":"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02","Type":"ContainerDied","Data":"c75d860b197105167fd2268f6657785086f4217b98146fe92871568d3d1930c5"} Oct 29 23:01:23.887161 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.886865 5610 generic.go:356] "Generic (PLEG): container finished" podID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerID="e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784" exitCode=0 Oct 29 23:01:23.887576 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.887162 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" event={"ID":"219cb1bb-3ba5-4b78-b457-0f4793cb4acf","Type":"ContainerDied","Data":"e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784"} Oct 29 23:01:23.887576 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.887212 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" event={"ID":"219cb1bb-3ba5-4b78-b457-0f4793cb4acf","Type":"ContainerDied","Data":"7b27eeb18be82a5afdca6fbe154e4bb50e62fe93751f7321c2789640e1981c14"} Oct 29 23:01:23.887576 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.887242 5610 scope.go:122] "RemoveContainer" containerID="e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784" Oct 29 23:01:23.887576 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.887421 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m" Oct 29 23:01:23.928362 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.925377 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.928362 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.925426 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.928362 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.925439 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.928362 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.925453 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgb74\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-kube-api-access-pgb74\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.928362 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.925469 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/219cb1bb-3ba5-4b78-b457-0f4793cb4acf-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:23.935557 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.933452 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:24.015307 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:23.992854 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m"] Oct 29 23:01:24.015307 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.000928 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-40cf9a-openshift-default-7449b86f87-4v55m"] Oct 29 23:01:24.020368 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.020141 5610 scope.go:122] "RemoveContainer" containerID="e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784" Oct 29 23:01:24.020880 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:24.020612 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784\": container with ID starting with e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784 not found: ID does not exist" containerID="e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784" Oct 29 23:01:24.020880 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.020658 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784"} err="failed to get container status \"e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784\": rpc error: code = NotFound desc = could not find container \"e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784\": container with ID starting with e6ab29fe83c3b0691a42ad4b5f2c33a1874394e9859535462db06dea363ad784 not found: ID does not exist" Oct 29 23:01:24.051900 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.051200 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 23:01:24.130508 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.129090 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-kube-api-access-t2vcb\" (UniqueName: \"kubernetes.io/projected/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-kube-api-access-t2vcb\") pod \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " Oct 29 23:01:24.130508 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.129228 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-metrics-certs\") pod \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " Oct 29 23:01:24.130508 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.129363 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-openshift-service-ca\") pod \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " Oct 29 23:01:24.130508 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.129584 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-maas-api-tls\") pod \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\" (UID: \"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02\") " Oct 29 23:01:24.152368 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.131307 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-openshift-service-ca" pod "8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" (UID: "8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:24.260427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.257852 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-kube-api-access-t2vcb" pod "8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" (UID: "8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02"). InnerVolumeSpecName "kube-api-access-t2vcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:24.260427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.259245 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2vcb\" (UniqueName: \"kubernetes.io/projected/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-kube-api-access-t2vcb\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.260427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.259275 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.342761 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.338596 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:24.381299 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.364718 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-metrics-certs" pod "8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" (UID: "8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:24.381299 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.366036 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.418433 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.417905 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-maas-api-tls" pod "8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" (UID: "8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:01:24.470894 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.469196 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.549107 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.549082 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 23:01:24.671537 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.671444 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-kube-api-access-4dd49\" (UniqueName: \"kubernetes.io/projected/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-kube-api-access-4dd49\") pod \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\" (UID: \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\") " Oct 29 23:01:24.671749 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.671726 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-plugins-config-volume\") pod \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\" (UID: \"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee\") " Oct 29 23:01:24.673242 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.673210 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-plugins-config-volume" pod "8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee" (UID: "8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:24.675161 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.675135 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-kube-api-access-4dd49" pod "8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee" (UID: "8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee"). InnerVolumeSpecName "kube-api-access-4dd49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:24.752639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.752600 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:24.774834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.774792 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dd49\" (UniqueName: \"kubernetes.io/projected/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-kube-api-access-4dd49\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.774834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.774832 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:24.899148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.899112 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" Oct 29 23:01:24.899148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.899136 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl" event={"ID":"8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02","Type":"ContainerDied","Data":"36b64f0a1cd4d4a30e3ae8b42736e499a5ce16abc15d455800cef2d8ccf4b490"} Oct 29 23:01:24.899793 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.899202 5610 scope.go:122] "RemoveContainer" containerID="c75d860b197105167fd2268f6657785086f4217b98146fe92871568d3d1930c5" Oct 29 23:01:24.901897 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.901858 5610 generic.go:356] "Generic (PLEG): container finished" podID="8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee" containerID="bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11" exitCode=0 Oct 29 23:01:24.902036 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.902017 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" Oct 29 23:01:24.902255 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.902231 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" event={"ID":"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee","Type":"ContainerDied","Data":"bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11"} Oct 29 23:01:24.902572 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.902547 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n" event={"ID":"8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee","Type":"ContainerDied","Data":"eca2fee5cce4e0513dbf8ea03a381316ab12d7952e4761a543abb0780966763e"} Oct 29 23:01:24.918651 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.918632 5610 scope.go:122] "RemoveContainer" containerID="bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11" Oct 29 23:01:24.929517 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.929498 5610 scope.go:122] "RemoveContainer" containerID="bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11" Oct 29 23:01:24.929887 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:24.929868 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11\": container with ID starting with bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11 not found: ID does not exist" containerID="bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11" Oct 29 23:01:24.929953 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.929896 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11"} err="failed to get container status \"bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11\": rpc error: code = NotFound desc = could not find container \"bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11\": container with ID starting with bc3eae620b3c06b418b79cc64ae68c066ee5a46be2c879dffd7ce934bfbace11 not found: ID does not exist" Oct 29 23:01:24.964404 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.964356 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n"] Oct 29 23:01:24.976610 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:24.976579 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bcb810-6c7bf88487-m8k5n"] Oct 29 23:01:25.030953 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:25.027331 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl"] Oct 29 23:01:25.034497 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:25.034456 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bcb810-5b9f84fc7d-tlbgl"] Oct 29 23:01:25.051857 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:25.051828 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:25.258689 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:25.258652 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:25.559561 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:25.559451 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:25.638827 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:25.638793 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" path="/var/lib/kubelet/pods/219cb1bb-3ba5-4b78-b457-0f4793cb4acf/volumes" Oct 29 23:01:25.639338 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:25.639321 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee" path="/var/lib/kubelet/pods/8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee/volumes" Oct 29 23:01:25.639786 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:25.639771 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" path="/var/lib/kubelet/pods/8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02/volumes" Oct 29 23:01:26.577723 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:26.577686 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:27.185327 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:27.185283 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:27.441717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:27.441636 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:27.445167 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:27.445140 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:01:27.445289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:27.445189 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:01:28.342114 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.342087 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508024 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-certs\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508161 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istiod-ca-cert\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508242 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-token\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-token\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508280 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-credential-socket\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508322 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-kube-api-access-7thd4\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-kube-api-access-7thd4\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508495 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-data\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508528 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-socket\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508584 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-envoy\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.508694 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-podinfo\") pod \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\" (UID: \"63f98200-dafd-4f5e-9440-a4e0c37a7b4a\") " Oct 29 23:01:28.510802 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.510318 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-data" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.510802 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.510568 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-certs" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.510802 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.510690 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-socket" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.510802 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.510699 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istiod-ca-cert" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:28.511012 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.510984 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-credential-socket" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.521552 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.521520 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-envoy" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.525913 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.525885 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-podinfo" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:28.526062 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.526042 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-token" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:28.528447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.528419 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-kube-api-access-7thd4" pod "63f98200-dafd-4f5e-9440-a4e0c37a7b4a" (UID: "63f98200-dafd-4f5e-9440-a4e0c37a7b4a"). InnerVolumeSpecName "kube-api-access-7thd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:28.611232 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611197 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.611232 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611232 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.611458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611247 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.611458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611278 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thd4\" (UniqueName: \"kubernetes.io/projected/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-kube-api-access-7thd4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.611458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611292 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.611458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611306 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.611458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611318 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.611458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611332 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.611458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.611343 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f98200-dafd-4f5e-9440-a4e0c37a7b4a-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:28.778264 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.778238 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:28.918122 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.917983 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-kube-api-access-hkvkc\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-kube-api-access-hkvkc\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.936045 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.936013 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-envoy\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.936481 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.936456 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-podinfo\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.936706 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.936689 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-token\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-token\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.944413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.938441 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-socket\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.944760 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.944732 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-certs\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.944992 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.944976 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-credential-socket\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.945151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.945134 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-data\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.945279 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.945264 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istiod-ca-cert\") pod \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\" (UID: \"5d34d23d-38d1-4525-bcc1-2a84d5c32724\") " Oct 29 23:01:28.956415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.951190 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-podinfo" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:28.956415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.951437 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-socket" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.956415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.951629 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-credential-socket" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.956415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.951832 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-data" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.956415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.953726 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-envoy" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:28.956952 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.956926 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istiod-ca-cert" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:29.003598 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:28.963470 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-certs" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:29.048829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.048682 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.048829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.048717 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.048829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.048731 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.048829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.048745 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.048829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.048763 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.048829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.048777 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.048829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.048790 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.049770 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.049736 5610 generic.go:356] "Generic (PLEG): container finished" podID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerID="8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8" exitCode=0 Oct 29 23:01:29.049938 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.049906 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" event={"ID":"5d34d23d-38d1-4525-bcc1-2a84d5c32724","Type":"ContainerDied","Data":"8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8"} Oct 29 23:01:29.050013 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.049970 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" event={"ID":"5d34d23d-38d1-4525-bcc1-2a84d5c32724","Type":"ContainerDied","Data":"b9223dd1f1c05bf55845474fd73f92fef7ee84881183082eba37591f666705cb"} Oct 29 23:01:29.050013 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.049998 5610 scope.go:122] "RemoveContainer" containerID="8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8" Oct 29 23:01:29.050151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.050136 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q" Oct 29 23:01:29.103458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.054657 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-token" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:29.175524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.131133 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-kube-api-access-hkvkc" pod "5d34d23d-38d1-4525-bcc1-2a84d5c32724" (UID: "5d34d23d-38d1-4525-bcc1-2a84d5c32724"). InnerVolumeSpecName "kube-api-access-hkvkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:29.175524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.150809 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkvkc\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-kube-api-access-hkvkc\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.175524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.150842 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d34d23d-38d1-4525-bcc1-2a84d5c32724-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:29.218412 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.214811 5610 generic.go:356] "Generic (PLEG): container finished" podID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerID="0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb" exitCode=0 Oct 29 23:01:29.218412 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.214911 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" event={"ID":"63f98200-dafd-4f5e-9440-a4e0c37a7b4a","Type":"ContainerDied","Data":"0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb"} Oct 29 23:01:29.218412 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.214974 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" event={"ID":"63f98200-dafd-4f5e-9440-a4e0c37a7b4a","Type":"ContainerDied","Data":"e8e26db2c419ef4d4f0c239805b3d5a6c6be1df86dbc6021ef1360a860860ba0"} Oct 29 23:01:29.218412 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.215069 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4" Oct 29 23:01:29.338638 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.337169 5610 scope.go:122] "RemoveContainer" containerID="8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8" Oct 29 23:01:29.342194 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:29.339589 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8\": container with ID starting with 8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8 not found: ID does not exist" containerID="8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8" Oct 29 23:01:29.342194 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.339628 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8"} err="failed to get container status \"8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8\": rpc error: code = NotFound desc = could not find container \"8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8\": container with ID starting with 8737c14acee94c7b75dd69df55b21461efe1ec2c9c1517b6291d73487228e2a8 not found: ID does not exist" Oct 29 23:01:29.342194 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.339656 5610 scope.go:122] "RemoveContainer" containerID="0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb" Oct 29 23:01:29.404553 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.404524 5610 scope.go:122] "RemoveContainer" containerID="0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566" Oct 29 23:01:29.408034 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.407792 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4"] Oct 29 23:01:29.408217 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.408203 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-xpll4"] Oct 29 23:01:29.422677 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.422650 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:29.451274 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.451253 5610 scope.go:122] "RemoveContainer" containerID="0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb" Oct 29 23:01:29.451831 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:29.451791 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb\": container with ID starting with 0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb not found: ID does not exist" containerID="0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb" Oct 29 23:01:29.452018 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.451986 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb"} err="failed to get container status \"0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb\": rpc error: code = NotFound desc = could not find container \"0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb\": container with ID starting with 0984535f942f943d9066fb2ec20a7ce2bea9bdaea2fe6192ab44caf209ebc7fb not found: ID does not exist" Oct 29 23:01:29.452128 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.452116 5610 scope.go:122] "RemoveContainer" containerID="0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566" Oct 29 23:01:29.453007 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:29.452986 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566\": container with ID starting with 0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566 not found: ID does not exist" containerID="0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566" Oct 29 23:01:29.453158 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.453136 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566"} err="failed to get container status \"0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566\": rpc error: code = NotFound desc = could not find container \"0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566\": container with ID starting with 0784ac505b793fab5d760120ac844fe400cd0d5747f792dad3f518b89a7c3566 not found: ID does not exist" Oct 29 23:01:29.458715 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.458658 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q"] Oct 29 23:01:29.466771 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.466661 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-329f7b-openshift-default-568bb78f87-mrq9q"] Oct 29 23:01:29.746470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.732951 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" path="/var/lib/kubelet/pods/5d34d23d-38d1-4525-bcc1-2a84d5c32724/volumes" Oct 29 23:01:29.746470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:29.733791 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" path="/var/lib/kubelet/pods/63f98200-dafd-4f5e-9440-a4e0c37a7b4a/volumes" Oct 29 23:01:30.706312 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.706276 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 23:01:30.759990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.759961 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 23:01:30.822622 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.822585 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8566790e-0736-4bbf-845f-3df7bbb82853-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8566790e-0736-4bbf-845f-3df7bbb82853-plugins-config-volume\") pod \"8566790e-0736-4bbf-845f-3df7bbb82853\" (UID: \"8566790e-0736-4bbf-845f-3df7bbb82853\") " Oct 29 23:01:30.822797 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.822668 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-plugins-config-volume\") pod \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\" (UID: \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\") " Oct 29 23:01:30.822797 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.822728 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-kube-api-access-j675w\" (UniqueName: \"kubernetes.io/projected/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-kube-api-access-j675w\") pod \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\" (UID: \"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec\") " Oct 29 23:01:30.822946 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.822923 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8566790e-0736-4bbf-845f-3df7bbb82853-kube-api-access-ng2t9\" (UniqueName: \"kubernetes.io/projected/8566790e-0736-4bbf-845f-3df7bbb82853-kube-api-access-ng2t9\") pod \"8566790e-0736-4bbf-845f-3df7bbb82853\" (UID: \"8566790e-0736-4bbf-845f-3df7bbb82853\") " Oct 29 23:01:30.823091 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.823041 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8566790e-0736-4bbf-845f-3df7bbb82853-plugins-config-volume" pod "8566790e-0736-4bbf-845f-3df7bbb82853" (UID: "8566790e-0736-4bbf-845f-3df7bbb82853"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:30.823684 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.823662 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8566790e-0736-4bbf-845f-3df7bbb82853-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.824258 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.824232 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-plugins-config-volume" pod "a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" (UID: "a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:30.825106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.825083 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8566790e-0736-4bbf-845f-3df7bbb82853-kube-api-access-ng2t9" pod "8566790e-0736-4bbf-845f-3df7bbb82853" (UID: "8566790e-0736-4bbf-845f-3df7bbb82853"). InnerVolumeSpecName "kube-api-access-ng2t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:30.827248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.827222 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-kube-api-access-j675w" pod "a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" (UID: "a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec"). InnerVolumeSpecName "kube-api-access-j675w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:30.924984 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.924911 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.924984 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.924938 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j675w\" (UniqueName: \"kubernetes.io/projected/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec-kube-api-access-j675w\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:30.924984 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:30.924949 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng2t9\" (UniqueName: \"kubernetes.io/projected/8566790e-0736-4bbf-845f-3df7bbb82853-kube-api-access-ng2t9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:31.234253 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.234217 5610 generic.go:356] "Generic (PLEG): container finished" podID="a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" containerID="06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378" exitCode=0 Oct 29 23:01:31.234472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.234261 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" Oct 29 23:01:31.234472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.234314 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" event={"ID":"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec","Type":"ContainerDied","Data":"06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378"} Oct 29 23:01:31.234472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.234429 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p" event={"ID":"a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec","Type":"ContainerDied","Data":"459e25ed9fffee6e3f03205f899247c8d6bce86fbbeba1590d2e28ede5e2c829"} Oct 29 23:01:31.234472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.234461 5610 scope.go:122] "RemoveContainer" containerID="06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378" Oct 29 23:01:31.236089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.236067 5610 generic.go:356] "Generic (PLEG): container finished" podID="8566790e-0736-4bbf-845f-3df7bbb82853" containerID="cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d" exitCode=0 Oct 29 23:01:31.236205 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.236117 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" Oct 29 23:01:31.236205 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.236112 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" event={"ID":"8566790e-0736-4bbf-845f-3df7bbb82853","Type":"ContainerDied","Data":"cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d"} Oct 29 23:01:31.236359 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.236245 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n" event={"ID":"8566790e-0736-4bbf-845f-3df7bbb82853","Type":"ContainerDied","Data":"cbae3127b6bb490e30727c834ce8196766b78bf1c69d4b1f16c5fcc0062201de"} Oct 29 23:01:31.246283 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.246266 5610 scope.go:122] "RemoveContainer" containerID="06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378" Oct 29 23:01:31.246568 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:31.246543 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378\": container with ID starting with 06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378 not found: ID does not exist" containerID="06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378" Oct 29 23:01:31.246636 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.246572 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378"} err="failed to get container status \"06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378\": rpc error: code = NotFound desc = could not find container \"06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378\": container with ID starting with 06df30f6db560dfd9e7241a5b82ee6b746d3284a0c87a34fd7898d0f7cb04378 not found: ID does not exist" Oct 29 23:01:31.246636 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.246588 5610 scope.go:122] "RemoveContainer" containerID="cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d" Oct 29 23:01:31.256144 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.256124 5610 scope.go:122] "RemoveContainer" containerID="cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d" Oct 29 23:01:31.256447 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:31.256387 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d\": container with ID starting with cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d not found: ID does not exist" containerID="cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d" Oct 29 23:01:31.256518 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.256459 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d"} err="failed to get container status \"cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d\": rpc error: code = NotFound desc = could not find container \"cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d\": container with ID starting with cd2ef68e43d0904f33d9883fbdb120cf8cd20f92669a57fe175f43abc8f0653d not found: ID does not exist" Oct 29 23:01:31.335380 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.335345 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p"] Oct 29 23:01:31.345105 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.345077 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-329f7b-6ddfd56fcb-vdg4p"] Oct 29 23:01:31.380739 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.380705 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n"] Oct 29 23:01:31.388566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.388542 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-329f7b-65cff495ff-tg24n"] Oct 29 23:01:31.638958 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.638872 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8566790e-0736-4bbf-845f-3df7bbb82853" path="/var/lib/kubelet/pods/8566790e-0736-4bbf-845f-3df7bbb82853/volumes" Oct 29 23:01:31.639308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.639293 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" path="/var/lib/kubelet/pods/a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec/volumes" Oct 29 23:01:31.804585 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.804554 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:31.966979 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:31.966919 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:32.143798 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:32.143749 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:32.509801 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:32.509766 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:32.852839 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:32.852764 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:01:33.977293 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:33.977252 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:34.137247 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:34.137208 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:34.400812 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:34.400733 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:35.809750 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:35.809709 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:37.174990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.174917 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2"] Oct 29 23:01:37.175898 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.175875 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" Oct 29 23:01:37.175898 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.175896 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.175949 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" containerName="maas-api" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.175955 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" containerName="maas-api" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.175972 5610 cpu_manager.go:414] "containerMap: removing container" podUID="a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" containerName="payload-processing" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.175977 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" containerName="payload-processing" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.175991 5610 cpu_manager.go:414] "containerMap: removing container" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.175999 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176013 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee" containerName="payload-pre-processing" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176018 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee" containerName="payload-pre-processing" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176026 5610 cpu_manager.go:414] "containerMap: removing container" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176031 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176042 5610 cpu_manager.go:414] "containerMap: removing container" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176046 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176053 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8566790e-0736-4bbf-845f-3df7bbb82853" containerName="payload-pre-processing" Oct 29 23:01:37.176067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176057 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8566790e-0736-4bbf-845f-3df7bbb82853" containerName="payload-pre-processing" Oct 29 23:01:37.176834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176179 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" Oct 29 23:01:37.176834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176189 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba0bcd7-3a49-4f4a-bfbf-6bfa5e6da1ee" containerName="payload-pre-processing" Oct 29 23:01:37.176834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176197 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="219cb1bb-3ba5-4b78-b457-0f4793cb4acf" containerName="istio-proxy" Oct 29 23:01:37.176834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176205 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8566790e-0736-4bbf-845f-3df7bbb82853" containerName="payload-pre-processing" Oct 29 23:01:37.176834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176214 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d34d23d-38d1-4525-bcc1-2a84d5c32724" containerName="istio-proxy" Oct 29 23:01:37.176834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176225 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dddf4d0-c8b0-4dc4-83b3-27d23aa54d02" containerName="maas-api" Oct 29 23:01:37.176834 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.176233 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b9e7a3-4d2a-4a52-8cc5-da01b5af79ec" containerName="payload-processing" Oct 29 23:01:37.181963 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.181940 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.205690 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.205662 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9ee1fc-serving-cert\"" Oct 29 23:01:37.211135 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.211110 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2"] Oct 29 23:01:37.305231 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.305195 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:37.351079 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.351035 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.351280 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.351091 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a68a161-97bd-429d-b6bb-28786285b6a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.351280 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.351236 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxcf\" (UniqueName: \"kubernetes.io/projected/3a68a161-97bd-429d-b6bb-28786285b6a9-kube-api-access-dtxcf\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.351529 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.351477 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.452566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.452524 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.452768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.452586 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.452768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.452651 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a68a161-97bd-429d-b6bb-28786285b6a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.452903 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.452779 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxcf\" (UniqueName: \"kubernetes.io/projected/3a68a161-97bd-429d-b6bb-28786285b6a9-kube-api-access-dtxcf\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.453371 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.453343 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a68a161-97bd-429d-b6bb-28786285b6a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.455179 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.455155 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.455295 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.455224 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.477965 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.477931 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:37.478736 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.478710 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxcf\" (UniqueName: \"kubernetes.io/projected/3a68a161-97bd-429d-b6bb-28786285b6a9-kube-api-access-dtxcf\") pod \"maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.510402 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.510353 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:37.646096 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.646061 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:37.843729 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:37.843703 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2"] Oct 29 23:01:37.844646 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:01:37.844618 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a68a161_97bd_429d_b6bb_28786285b6a9.slice/crio-dacc3b9c9ac3c8bd99c306ac0a3b2a7906ffbf6a76004899dc78a207074654cd WatchSource:0}: Error finding container dacc3b9c9ac3c8bd99c306ac0a3b2a7906ffbf6a76004899dc78a207074654cd: Status 404 returned error can't find the container with id dacc3b9c9ac3c8bd99c306ac0a3b2a7906ffbf6a76004899dc78a207074654cd Oct 29 23:01:38.149491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:38.149411 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v"] Oct 29 23:01:38.149715 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:38.149671 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" containerID="cri-o://8a17ba0e840e65dc1edb3893359d8cef900b35c8334efdbf29b72390e68b496c" gracePeriod=30 Oct 29 23:01:38.459951 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:38.455387 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" event={"ID":"3a68a161-97bd-429d-b6bb-28786285b6a9","Type":"ContainerStarted","Data":"dacc3b9c9ac3c8bd99c306ac0a3b2a7906ffbf6a76004899dc78a207074654cd"} Oct 29 23:01:39.137418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:39.137040 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:39.465169 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:39.465130 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" event={"ID":"3a68a161-97bd-429d-b6bb-28786285b6a9","Type":"ContainerStarted","Data":"812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e"} Oct 29 23:01:39.465730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:39.465282 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:39.508614 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:39.508559 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" podStartSLOduration=2.015381172 podStartE2EDuration="2.508544908s" totalImagesPullingTime="493.163736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:39.504919715 +0000 UTC m=+2761.986237209" watchObservedRunningTime="2026-10-29 23:01:39.508544908 +0000 UTC m=+2761.989862434" Oct 29 23:01:40.664470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:40.663612 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:40.813737 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:40.813698 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:42.178412 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:42.177547 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:42.970996 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:42.970963 5610 scope.go:122] "RemoveContainer" containerID="7ac6eeae2cd04bb269b859c0d62d0e9821ea039053b9cfd2c87d94c758baf8b8" Oct 29 23:01:43.113744 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.113707 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl"] Oct 29 23:01:43.348618 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.348586 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:43.509583 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.509085 5610 generic.go:356] "Generic (PLEG): container finished" podID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerID="8a17ba0e840e65dc1edb3893359d8cef900b35c8334efdbf29b72390e68b496c" exitCode=0 Oct 29 23:01:43.509583 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.509236 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" event={"ID":"8f49e776-3db8-439c-9b4f-fa685c3b8fa3","Type":"ContainerDied","Data":"8a17ba0e840e65dc1edb3893359d8cef900b35c8334efdbf29b72390e68b496c"} Oct 29 23:01:43.523122 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.523091 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:43.542197 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.542136 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 23:01:43.666951 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.666906 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-podinfo\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667084 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-token\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-token\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667119 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-data\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667451 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667280 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-kube-api-access-5kqcr\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-kube-api-access-5kqcr\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667451 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667319 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-credential-socket\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667451 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667356 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istiod-ca-cert\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667451 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667438 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-certs\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667762 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667482 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-socket\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667762 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667518 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-envoy\") pod \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\" (UID: \"8f49e776-3db8-439c-9b4f-fa685c3b8fa3\") " Oct 29 23:01:43.667915 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.667862 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-socket" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:43.668438 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.668373 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:43.668699 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.668569 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-certs" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:43.668836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.668811 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-credential-socket" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:43.669507 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.669482 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-data" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:43.669722 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.669700 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istiod-ca-cert" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:01:43.671655 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.671628 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-envoy" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:01:43.671947 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.671924 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-token" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:43.673746 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.673724 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-kube-api-access-5kqcr" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "kube-api-access-5kqcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:01:43.674325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.674294 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-podinfo" pod "8f49e776-3db8-439c-9b4f-fa685c3b8fa3" (UID: "8f49e776-3db8-439c-9b4f-fa685c3b8fa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:01:43.719730 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:01:43.719692 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:01:43.769557 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.769528 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kqcr\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-kube-api-access-5kqcr\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:43.769703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.769573 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:43.769703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.769585 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:43.769703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.769595 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:43.769703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.769604 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:43.769703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.769611 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:43.769703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.769619 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:43.769703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:43.769627 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49e776-3db8-439c-9b4f-fa685c3b8fa3-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:01:44.047418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:44.045139 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:44.520193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:44.520164 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" Oct 29 23:01:44.520193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:44.520183 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v" event={"ID":"8f49e776-3db8-439c-9b4f-fa685c3b8fa3","Type":"ContainerDied","Data":"3f1c9c22c19559a1a67ea57e8150856465c760b4e7951e396a58858daf9d0c29"} Oct 29 23:01:44.520795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:44.520243 5610 scope.go:122] "RemoveContainer" containerID="8a17ba0e840e65dc1edb3893359d8cef900b35c8334efdbf29b72390e68b496c" Oct 29 23:01:44.586457 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:44.586427 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v"] Oct 29 23:01:44.589727 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:44.589698 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6bb06a-openshift-default-df9874c47-msc5v"] Oct 29 23:01:45.480949 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:45.480922 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:01:45.648062 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:45.648015 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" path="/var/lib/kubelet/pods/8f49e776-3db8-439c-9b4f-fa685c3b8fa3/volumes" Oct 29 23:01:45.802927 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:45.802842 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:47.104420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:47.104355 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:47.729961 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:47.729916 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:47.931758 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:47.931720 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:48.114998 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:48.114891 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" podUID="bd36ddb4-7658-4597-a881-a9c36e7fb289" containerName="payload-processing" containerID="cri-o://14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5" gracePeriod=25 Oct 29 23:01:48.494574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:48.494542 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:49.795999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.795966 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt"] Oct 29 23:01:49.796972 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.796949 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" Oct 29 23:01:49.796972 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.796970 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" Oct 29 23:01:49.797143 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.797117 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f49e776-3db8-439c-9b4f-fa685c3b8fa3" containerName="istio-proxy" Oct 29 23:01:49.797143 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.797129 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f98200-dafd-4f5e-9440-a4e0c37a7b4a" containerName="istio-proxy" Oct 29 23:01:49.803021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.803001 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:49.826719 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.826690 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3e2b2c-serving-cert\"" Oct 29 23:01:49.840948 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.840910 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt"] Oct 29 23:01:49.942979 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.942948 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7jj\" (UniqueName: \"kubernetes.io/projected/caeab242-8754-4bee-8f24-41b26ed86f2d-kube-api-access-ln7jj\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:49.942979 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.942990 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:49.943206 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.943147 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:49.943290 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:49.943269 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeab242-8754-4bee-8f24-41b26ed86f2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.043245 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.043206 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:50.044681 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.044652 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.044820 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.044767 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeab242-8754-4bee-8f24-41b26ed86f2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.044909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.044892 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7jj\" (UniqueName: \"kubernetes.io/projected/caeab242-8754-4bee-8f24-41b26ed86f2d-kube-api-access-ln7jj\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.044981 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.044933 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.045572 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.045533 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeab242-8754-4bee-8f24-41b26ed86f2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.047344 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.047295 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.047680 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.047656 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.092292 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.092257 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7jj\" (UniqueName: \"kubernetes.io/projected/caeab242-8754-4bee-8f24-41b26ed86f2d-kube-api-access-ln7jj\") pod \"maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.121077 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.121034 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:50.425331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.425245 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:50.443172 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.443145 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt"] Oct 29 23:01:50.443750 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:01:50.443717 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaeab242_8754_4bee_8f24_41b26ed86f2d.slice/crio-8c2d3911dc6ef919dafb3fc4948131da7c83d924059459c62a0fe489e4a39576 WatchSource:0}: Error finding container 8c2d3911dc6ef919dafb3fc4948131da7c83d924059459c62a0fe489e4a39576: Status 404 returned error can't find the container with id 8c2d3911dc6ef919dafb3fc4948131da7c83d924059459c62a0fe489e4a39576 Oct 29 23:01:50.577354 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:50.577310 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" event={"ID":"caeab242-8754-4bee-8f24-41b26ed86f2d","Type":"ContainerStarted","Data":"8c2d3911dc6ef919dafb3fc4948131da7c83d924059459c62a0fe489e4a39576"} Oct 29 23:01:51.586355 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:51.586319 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" event={"ID":"caeab242-8754-4bee-8f24-41b26ed86f2d","Type":"ContainerStarted","Data":"364758c30f75d090998d2eb9975ef65d1b09f48709fb64e5569389e2f6c93c38"} Oct 29 23:01:51.586783 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:51.586427 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:51.644864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:51.644808 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" podStartSLOduration=2.1882181 podStartE2EDuration="2.644793955s" totalImagesPullingTime="456.575855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:01:51.643289263 +0000 UTC m=+2774.124606756" watchObservedRunningTime="2026-10-29 23:01:51.644793955 +0000 UTC m=+2774.126111441" Oct 29 23:01:52.580539 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:52.580501 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:54.467487 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:54.463186 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:56.426907 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:56.426858 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:56.728708 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:56.728674 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:56.916356 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:56.916314 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" podUID="bd36ddb4-7658-4597-a881-a9c36e7fb289" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:01:57.002217 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:57.002126 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:01:57.600069 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:57.600038 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:01:58.724577 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:01:58.724530 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:00.376199 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:00.376154 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:02.006475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:02.002587 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:03.435031 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:03.434988 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:04.903917 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:04.903875 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:06.514794 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:06.514758 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4"] Oct 29 23:02:06.515200 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:06.514989 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" podUID="f91681c3-825b-4be3-a29a-6231db8b969a" containerName="maas-api" containerID="cri-o://a60e0bd80a011f02390210ba649e4d2e3b496b1968a944544805eca3346f9b86" gracePeriod=30 Oct 29 23:02:06.617949 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:06.617909 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:06.784735 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:06.779140 5610 generic.go:356] "Generic (PLEG): container finished" podID="f91681c3-825b-4be3-a29a-6231db8b969a" containerID="a60e0bd80a011f02390210ba649e4d2e3b496b1968a944544805eca3346f9b86" exitCode=0 Oct 29 23:02:06.784735 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:06.779478 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" event={"ID":"f91681c3-825b-4be3-a29a-6231db8b969a","Type":"ContainerDied","Data":"a60e0bd80a011f02390210ba649e4d2e3b496b1968a944544805eca3346f9b86"} Oct 29 23:02:06.991426 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:06.980582 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" podUID="bd36ddb4-7658-4597-a881-a9c36e7fb289" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:07.013701 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.013667 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 23:02:07.129026 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.128925 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91681c3-825b-4be3-a29a-6231db8b969a-kube-api-access-8ll5r\" (UniqueName: \"kubernetes.io/projected/f91681c3-825b-4be3-a29a-6231db8b969a-kube-api-access-8ll5r\") pod \"f91681c3-825b-4be3-a29a-6231db8b969a\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " Oct 29 23:02:07.129211 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.129142 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-maas-api-tls\") pod \"f91681c3-825b-4be3-a29a-6231db8b969a\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " Oct 29 23:02:07.129281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.129209 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-metrics-certs\") pod \"f91681c3-825b-4be3-a29a-6231db8b969a\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " Oct 29 23:02:07.129903 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.129483 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91681c3-825b-4be3-a29a-6231db8b969a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91681c3-825b-4be3-a29a-6231db8b969a-openshift-service-ca\") pod \"f91681c3-825b-4be3-a29a-6231db8b969a\" (UID: \"f91681c3-825b-4be3-a29a-6231db8b969a\") " Oct 29 23:02:07.129903 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.129852 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91681c3-825b-4be3-a29a-6231db8b969a-openshift-service-ca" pod "f91681c3-825b-4be3-a29a-6231db8b969a" (UID: "f91681c3-825b-4be3-a29a-6231db8b969a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:07.130275 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.130249 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f91681c3-825b-4be3-a29a-6231db8b969a-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:07.131690 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.131663 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91681c3-825b-4be3-a29a-6231db8b969a-kube-api-access-8ll5r" pod "f91681c3-825b-4be3-a29a-6231db8b969a" (UID: "f91681c3-825b-4be3-a29a-6231db8b969a"). InnerVolumeSpecName "kube-api-access-8ll5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:07.132079 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.132054 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-maas-api-tls" pod "f91681c3-825b-4be3-a29a-6231db8b969a" (UID: "f91681c3-825b-4be3-a29a-6231db8b969a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:07.132284 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.132253 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-metrics-certs" pod "f91681c3-825b-4be3-a29a-6231db8b969a" (UID: "f91681c3-825b-4be3-a29a-6231db8b969a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:02:07.196058 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.196019 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:07.231627 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.231590 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ll5r\" (UniqueName: \"kubernetes.io/projected/f91681c3-825b-4be3-a29a-6231db8b969a-kube-api-access-8ll5r\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:07.231627 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.231623 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:07.231627 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.231633 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f91681c3-825b-4be3-a29a-6231db8b969a-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:07.385888 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.385774 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:07.549111 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.549074 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:07.789350 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.789321 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" Oct 29 23:02:07.789350 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.789345 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4" event={"ID":"f91681c3-825b-4be3-a29a-6231db8b969a","Type":"ContainerDied","Data":"d90bbe44dd199241e2ac3cb37dde5b5ba41181dcccc6833e1f0fb7471522e0b2"} Oct 29 23:02:07.789605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.789428 5610 scope.go:122] "RemoveContainer" containerID="a60e0bd80a011f02390210ba649e4d2e3b496b1968a944544805eca3346f9b86" Oct 29 23:02:07.844493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.844465 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4"] Oct 29 23:02:07.847873 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.847840 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z"] Oct 29 23:02:07.855792 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.855764 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e803b6-5c9d54cccc-tv2h4"] Oct 29 23:02:07.948532 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:07.948291 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:08.152679 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.150190 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:08.424801 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.424775 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 23:02:08.454446 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.454405 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd36ddb4-7658-4597-a881-a9c36e7fb289-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36ddb4-7658-4597-a881-a9c36e7fb289-plugins-config-volume\") pod \"bd36ddb4-7658-4597-a881-a9c36e7fb289\" (UID: \"bd36ddb4-7658-4597-a881-a9c36e7fb289\") " Oct 29 23:02:08.454620 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.454515 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd36ddb4-7658-4597-a881-a9c36e7fb289-kube-api-access-gdwt2\" (UniqueName: \"kubernetes.io/projected/bd36ddb4-7658-4597-a881-a9c36e7fb289-kube-api-access-gdwt2\") pod \"bd36ddb4-7658-4597-a881-a9c36e7fb289\" (UID: \"bd36ddb4-7658-4597-a881-a9c36e7fb289\") " Oct 29 23:02:08.454807 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.454780 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd36ddb4-7658-4597-a881-a9c36e7fb289-plugins-config-volume" pod "bd36ddb4-7658-4597-a881-a9c36e7fb289" (UID: "bd36ddb4-7658-4597-a881-a9c36e7fb289"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:08.455209 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.455189 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd36ddb4-7658-4597-a881-a9c36e7fb289-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:08.456891 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.456871 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd36ddb4-7658-4597-a881-a9c36e7fb289-kube-api-access-gdwt2" pod "bd36ddb4-7658-4597-a881-a9c36e7fb289" (UID: "bd36ddb4-7658-4597-a881-a9c36e7fb289"). InnerVolumeSpecName "kube-api-access-gdwt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:08.557113 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.557031 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdwt2\" (UniqueName: \"kubernetes.io/projected/bd36ddb4-7658-4597-a881-a9c36e7fb289-kube-api-access-gdwt2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:08.819467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.813744 5610 generic.go:356] "Generic (PLEG): container finished" podID="bd36ddb4-7658-4597-a881-a9c36e7fb289" containerID="14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5" exitCode=0 Oct 29 23:02:08.819467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.813947 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" event={"ID":"bd36ddb4-7658-4597-a881-a9c36e7fb289","Type":"ContainerDied","Data":"14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5"} Oct 29 23:02:08.819467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.814001 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" event={"ID":"bd36ddb4-7658-4597-a881-a9c36e7fb289","Type":"ContainerDied","Data":"f15ee260857a7b545b21954710d70a37cdfe078e595a4837055e05e4f1d07b3d"} Oct 29 23:02:08.819467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.814032 5610 scope.go:122] "RemoveContainer" containerID="14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5" Oct 29 23:02:08.819467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.814138 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl" Oct 29 23:02:08.833900 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.833876 5610 scope.go:122] "RemoveContainer" containerID="14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5" Oct 29 23:02:08.834435 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:02:08.834184 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5\": container with ID starting with 14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5 not found: ID does not exist" containerID="14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5" Oct 29 23:02:08.834435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.834226 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5"} err="failed to get container status \"14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5\": rpc error: code = NotFound desc = could not find container \"14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5\": container with ID starting with 14edf8fef338a22d75d4306d89ef9b9b640463bef7d8b35a904423affe3056d5 not found: ID does not exist" Oct 29 23:02:08.870199 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.870164 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl"] Oct 29 23:02:08.878966 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:08.878938 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-afdbe8-78bf8756d4-h68rl"] Oct 29 23:02:09.638434 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:09.638373 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd36ddb4-7658-4597-a881-a9c36e7fb289" path="/var/lib/kubelet/pods/bd36ddb4-7658-4597-a881-a9c36e7fb289/volumes" Oct 29 23:02:09.638894 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:09.638878 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91681c3-825b-4be3-a29a-6231db8b969a" path="/var/lib/kubelet/pods/f91681c3-825b-4be3-a29a-6231db8b969a/volumes" Oct 29 23:02:10.384420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:10.384005 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:11.233172 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:11.232084 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:12.622874 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:12.622832 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph"] Oct 29 23:02:12.624854 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:12.624828 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6"] Oct 29 23:02:12.830062 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:12.829974 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:12.885908 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:12.882517 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" podUID="28642a96-d2b2-40e5-b32e-7fa96f97efe9" containerName="payload-processing" containerID="cri-o://263cfeb7a14bd60d79aefbb844c06c99407788dbc87a39f41074abe7050f6d4a" gracePeriod=25 Oct 29 23:02:13.081968 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:13.052907 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:13.749909 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:02:13.749870 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:02:14.256283 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:14.256205 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:14.426516 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:14.426155 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:14.541741 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:14.541651 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b"] Oct 29 23:02:14.542052 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:14.541986 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" containerID="cri-o://56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21" gracePeriod=30 Oct 29 23:02:14.645381 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:14.645343 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:14.991022 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:14.990986 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:16.255706 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:16.255638 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:17.623630 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:17.623582 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" podUID="9aead210-f299-4300-9978-f24becb36fed" containerName="payload-processing" containerID="cri-o://a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8" gracePeriod=25 Oct 29 23:02:17.625789 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:17.625612 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" podUID="442517be-d8d2-4a32-b0c4-255bb0934adc" containerName="payload-pre-processing" containerID="cri-o://ba9c67c795f8115c3a53f33508e57815b5b694df6843ad895d657ab633a7f86c" gracePeriod=25 Oct 29 23:02:18.003593 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.003211 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:02:18.061052 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.060830 5610 generic.go:356] "Generic (PLEG): container finished" podID="9aead210-f299-4300-9978-f24becb36fed" containerID="a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8" exitCode=0 Oct 29 23:02:18.061052 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.060896 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" event={"ID":"9aead210-f299-4300-9978-f24becb36fed","Type":"ContainerDied","Data":"a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8"} Oct 29 23:02:18.061052 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.060964 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" event={"ID":"9aead210-f299-4300-9978-f24becb36fed","Type":"ContainerDied","Data":"29fb8577addfdcc42e636dc01e8644909e02ebb04dad4e90c1bcda6bc75e7a19"} Oct 29 23:02:18.061052 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.060999 5610 scope.go:122] "RemoveContainer" containerID="a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8" Oct 29 23:02:18.062045 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.062024 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph" Oct 29 23:02:18.064777 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.064485 5610 generic.go:356] "Generic (PLEG): container finished" podID="442517be-d8d2-4a32-b0c4-255bb0934adc" containerID="ba9c67c795f8115c3a53f33508e57815b5b694df6843ad895d657ab633a7f86c" exitCode=0 Oct 29 23:02:18.064777 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.064628 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" event={"ID":"442517be-d8d2-4a32-b0c4-255bb0934adc","Type":"ContainerDied","Data":"ba9c67c795f8115c3a53f33508e57815b5b694df6843ad895d657ab633a7f86c"} Oct 29 23:02:18.077190 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.077099 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aead210-f299-4300-9978-f24becb36fed-kube-api-access-q9857\" (UniqueName: \"kubernetes.io/projected/9aead210-f299-4300-9978-f24becb36fed-kube-api-access-q9857\") pod \"9aead210-f299-4300-9978-f24becb36fed\" (UID: \"9aead210-f299-4300-9978-f24becb36fed\") " Oct 29 23:02:18.078876 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.078858 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aead210-f299-4300-9978-f24becb36fed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aead210-f299-4300-9978-f24becb36fed-plugins-config-volume\") pod \"9aead210-f299-4300-9978-f24becb36fed\" (UID: \"9aead210-f299-4300-9978-f24becb36fed\") " Oct 29 23:02:18.079301 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.079273 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aead210-f299-4300-9978-f24becb36fed-plugins-config-volume" pod "9aead210-f299-4300-9978-f24becb36fed" (UID: "9aead210-f299-4300-9978-f24becb36fed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:18.080094 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.080075 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aead210-f299-4300-9978-f24becb36fed-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:18.081697 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.081661 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aead210-f299-4300-9978-f24becb36fed-kube-api-access-q9857" pod "9aead210-f299-4300-9978-f24becb36fed" (UID: "9aead210-f299-4300-9978-f24becb36fed"). InnerVolumeSpecName "kube-api-access-q9857". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:18.106434 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.104550 5610 scope.go:122] "RemoveContainer" containerID="a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8" Oct 29 23:02:18.106771 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:02:18.106677 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8\": container with ID starting with a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8 not found: ID does not exist" containerID="a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8" Oct 29 23:02:18.106771 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.106724 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8"} err="failed to get container status \"a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8\": rpc error: code = NotFound desc = could not find container \"a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8\": container with ID starting with a795c962ed0ef8667793401b269d34e0308823cf053b097e56fbf9bb6cfbcdd8 not found: ID does not exist" Oct 29 23:02:18.127352 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.127333 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:02:18.181220 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.181179 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442517be-d8d2-4a32-b0c4-255bb0934adc-kube-api-access-4r5b5\" (UniqueName: \"kubernetes.io/projected/442517be-d8d2-4a32-b0c4-255bb0934adc-kube-api-access-4r5b5\") pod \"442517be-d8d2-4a32-b0c4-255bb0934adc\" (UID: \"442517be-d8d2-4a32-b0c4-255bb0934adc\") " Oct 29 23:02:18.181468 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.181340 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442517be-d8d2-4a32-b0c4-255bb0934adc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/442517be-d8d2-4a32-b0c4-255bb0934adc-plugins-config-volume\") pod \"442517be-d8d2-4a32-b0c4-255bb0934adc\" (UID: \"442517be-d8d2-4a32-b0c4-255bb0934adc\") " Oct 29 23:02:18.182018 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.181981 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9857\" (UniqueName: \"kubernetes.io/projected/9aead210-f299-4300-9978-f24becb36fed-kube-api-access-q9857\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:18.182459 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.182435 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442517be-d8d2-4a32-b0c4-255bb0934adc-plugins-config-volume" pod "442517be-d8d2-4a32-b0c4-255bb0934adc" (UID: "442517be-d8d2-4a32-b0c4-255bb0934adc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:18.185345 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.185316 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442517be-d8d2-4a32-b0c4-255bb0934adc-kube-api-access-4r5b5" pod "442517be-d8d2-4a32-b0c4-255bb0934adc" (UID: "442517be-d8d2-4a32-b0c4-255bb0934adc"). InnerVolumeSpecName "kube-api-access-4r5b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:18.271539 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.271355 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:18.283709 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.283635 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5b5\" (UniqueName: \"kubernetes.io/projected/442517be-d8d2-4a32-b0c4-255bb0934adc-kube-api-access-4r5b5\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:18.283709 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.283672 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/442517be-d8d2-4a32-b0c4-255bb0934adc-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:18.356465 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.356425 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:02:18.356676 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.356486 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:02:18.416523 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.416486 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph"] Oct 29 23:02:18.423776 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:18.423744 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-74d89cb4c8-dppph"] Oct 29 23:02:19.077410 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:19.077350 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" Oct 29 23:02:19.077410 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:19.077345 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6" event={"ID":"442517be-d8d2-4a32-b0c4-255bb0934adc","Type":"ContainerDied","Data":"e0d233d14b1f19ac00fd84b1bd18dad85539a1bf513675f886dbb8a630837f20"} Oct 29 23:02:19.078550 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:19.077466 5610 scope.go:122] "RemoveContainer" containerID="ba9c67c795f8115c3a53f33508e57815b5b694df6843ad895d657ab633a7f86c" Oct 29 23:02:19.134952 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:19.134914 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6"] Oct 29 23:02:19.138285 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:19.138256 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-6ffff46bf5-4xkd6"] Oct 29 23:02:19.649615 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:19.649580 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442517be-d8d2-4a32-b0c4-255bb0934adc" path="/var/lib/kubelet/pods/442517be-d8d2-4a32-b0c4-255bb0934adc/volumes" Oct 29 23:02:19.650181 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:19.650157 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aead210-f299-4300-9978-f24becb36fed" path="/var/lib/kubelet/pods/9aead210-f299-4300-9978-f24becb36fed/volumes" Oct 29 23:02:19.910532 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:19.910140 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.017855 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-envoy\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.017995 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-socket\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.018039 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-data\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.018105 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89be13e4-0548-49e6-8c81-006939d66752-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be13e4-0548-49e6-8c81-006939d66752-istio-podinfo\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.018158 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-kube-api-access-sh78m\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-kube-api-access-sh78m\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.018251 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89be13e4-0548-49e6-8c81-006939d66752-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be13e4-0548-49e6-8c81-006939d66752-istiod-ca-cert\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.018304 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-credential-socket\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.018474 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-certs\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.018937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.018531 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-istio-token\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-istio-token\") pod \"89be13e4-0548-49e6-8c81-006939d66752\" (UID: \"89be13e4-0548-49e6-8c81-006939d66752\") " Oct 29 23:02:20.020292 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.020049 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-socket" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:20.021350 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.021246 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-data" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:20.021350 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.021292 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89be13e4-0548-49e6-8c81-006939d66752-istiod-ca-cert" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:20.021615 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.021521 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-credential-socket" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:20.021961 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.021881 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.021961 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.021909 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.021961 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.021924 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89be13e4-0548-49e6-8c81-006939d66752-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.021961 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.021936 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.022236 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.022115 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-certs" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:20.024104 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.024072 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-istio-token" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:20.026634 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.026606 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-envoy" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:20.027970 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.027927 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-kube-api-access-sh78m" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "kube-api-access-sh78m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:20.028953 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.028926 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89be13e4-0548-49e6-8c81-006939d66752-istio-podinfo" pod "89be13e4-0548-49e6-8c81-006939d66752" (UID: "89be13e4-0548-49e6-8c81-006939d66752"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:20.094686 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.094650 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:20.097920 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.097883 5610 generic.go:356] "Generic (PLEG): container finished" podID="89be13e4-0548-49e6-8c81-006939d66752" containerID="56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21" exitCode=0 Oct 29 23:02:20.098175 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.098156 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" Oct 29 23:02:20.100631 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.100598 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" event={"ID":"89be13e4-0548-49e6-8c81-006939d66752","Type":"ContainerDied","Data":"56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21"} Oct 29 23:02:20.100758 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.100660 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b" event={"ID":"89be13e4-0548-49e6-8c81-006939d66752","Type":"ContainerDied","Data":"00347ebcfcaecc1b94df2133645b8a5faaf0b6d6b43c573bdc9f00f3b17a51f5"} Oct 29 23:02:20.100758 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.100690 5610 scope.go:122] "RemoveContainer" containerID="56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21" Oct 29 23:02:20.118052 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.118033 5610 scope.go:122] "RemoveContainer" containerID="28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9" Oct 29 23:02:20.123263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.123244 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.123263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.123265 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.123453 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.123273 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89be13e4-0548-49e6-8c81-006939d66752-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.123453 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.123281 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89be13e4-0548-49e6-8c81-006939d66752-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.123453 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.123289 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh78m\" (UniqueName: \"kubernetes.io/projected/89be13e4-0548-49e6-8c81-006939d66752-kube-api-access-sh78m\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:20.129507 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.129482 5610 scope.go:122] "RemoveContainer" containerID="56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21" Oct 29 23:02:20.129845 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:02:20.129818 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21\": container with ID starting with 56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21 not found: ID does not exist" containerID="56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21" Oct 29 23:02:20.129925 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.129856 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21"} err="failed to get container status \"56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21\": rpc error: code = NotFound desc = could not find container \"56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21\": container with ID starting with 56ee2e55c4c86b6a8bde7d8c8e02c374d04ac2e70040b12a18f58cddbbd85d21 not found: ID does not exist" Oct 29 23:02:20.130202 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.129880 5610 scope.go:122] "RemoveContainer" containerID="28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9" Oct 29 23:02:20.130479 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:02:20.130453 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9\": container with ID starting with 28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9 not found: ID does not exist" containerID="28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9" Oct 29 23:02:20.130556 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.130489 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9"} err="failed to get container status \"28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9\": rpc error: code = NotFound desc = could not find container \"28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9\": container with ID starting with 28f5b2005b339ece0aafe29c553fa4fec7651ad4708dc1085ea803253b11b6a9 not found: ID does not exist" Oct 29 23:02:20.155855 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.155828 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b"] Oct 29 23:02:20.163260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.163229 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e803b6-openshift-default-79f8f89c4c-jrg8b"] Oct 29 23:02:20.254898 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.254865 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:20.484447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.483355 5610 patch_prober.go:29] interesting pod/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": read tcp 10.134.0.2:52806->10.134.0.233:15021: read: connection reset by peer" start-of-body= Oct 29 23:02:20.484447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:20.483429 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": read tcp 10.134.0.2:52806->10.134.0.233:15021: read: connection reset by peer" Oct 29 23:02:21.111615 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.111535 5610 generic.go:356] "Generic (PLEG): container finished" podID="e77602a7-cf53-413f-9484-7baccce25ea5" containerID="d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa" exitCode=0 Oct 29 23:02:21.111615 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.111603 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" event={"ID":"e77602a7-cf53-413f-9484-7baccce25ea5","Type":"ContainerDied","Data":"d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa"} Oct 29 23:02:21.112157 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.112078 5610 scope.go:122] "RemoveContainer" containerID="d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa" Oct 29 23:02:21.113461 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.113434 5610 generic.go:356] "Generic (PLEG): container finished" podID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerID="44d98ba7967fcd6dfc9c63a27f14cee4be9ee087e3153fdc061607c23f1f071e" exitCode=0 Oct 29 23:02:21.113594 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.113506 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" event={"ID":"0e83482a-0401-47df-ab5a-1b419b3c371d","Type":"ContainerDied","Data":"44d98ba7967fcd6dfc9c63a27f14cee4be9ee087e3153fdc061607c23f1f071e"} Oct 29 23:02:21.113886 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.113868 5610 scope.go:122] "RemoveContainer" containerID="44d98ba7967fcd6dfc9c63a27f14cee4be9ee087e3153fdc061607c23f1f071e" Oct 29 23:02:21.114024 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.113998 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:21.114107 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.114089 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:21.114173 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.114147 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:21.115548 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.115497 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:21.115659 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.115601 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:21.115728 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.115671 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:21.219935 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.219822 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" podUID="28642a96-d2b2-40e5-b32e-7fa96f97efe9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:21.655546 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:21.654096 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89be13e4-0548-49e6-8c81-006939d66752" path="/var/lib/kubelet/pods/89be13e4-0548-49e6-8c81-006939d66752/volumes" Oct 29 23:02:22.003423 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:22.000662 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:22.131431 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:22.130834 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" event={"ID":"e77602a7-cf53-413f-9484-7baccce25ea5","Type":"ContainerStarted","Data":"a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def"} Oct 29 23:02:22.136271 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:22.136235 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" event={"ID":"0e83482a-0401-47df-ab5a-1b419b3c371d","Type":"ContainerStarted","Data":"31a0f05c93d788d20ca3c1cc6e8c8ea693b48bb673437ace892029712a8e9725"} Oct 29 23:02:22.855646 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:22.855553 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:02:22.855932 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:22.855907 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 29 23:02:22.856001 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:22.855969 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 29 23:02:23.030953 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:23.030849 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:02:23.032487 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:23.032384 5610 patch_prober.go:29] interesting pod/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 23:02:23.032487 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:23.032468 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 23:02:23.558914 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:23.558859 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:23.846910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:23.846783 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:23.863420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:23.858534 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 29 23:02:23.863420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:23.858591 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 29 23:02:24.047417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:24.046372 5610 patch_prober.go:29] interesting pod/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 23:02:24.047417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:24.046474 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 23:02:24.171513 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:24.167643 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:24.855739 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:24.855706 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 29 23:02:24.856190 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:24.856163 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 29 23:02:25.023419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:25.020909 5610 patch_prober.go:29] interesting pod/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 23:02:25.023419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:25.020970 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 23:02:25.783424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:25.782283 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:25.857326 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:25.856191 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 29 23:02:25.857326 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:25.856249 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 29 23:02:25.930723 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:25.930150 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:26.022083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:26.022020 5610 patch_prober.go:29] interesting pod/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 23:02:26.022469 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:26.022438 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 23:02:26.855923 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:26.855854 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 29 23:02:26.855923 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:26.855909 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 29 23:02:27.020424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:27.019838 5610 patch_prober.go:29] interesting pod/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 23:02:27.020424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:27.019911 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 23:02:27.855318 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:27.855279 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Oct 29 23:02:27.855575 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:27.855346 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Oct 29 23:02:27.932412 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:27.930119 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:28.020304 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:28.020257 5610 patch_prober.go:29] interesting pod/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Oct 29 23:02:28.020549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:28.020324 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Oct 29 23:02:28.859999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:28.859965 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:02:28.860421 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:28.860363 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:02:28.861085 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:28.861062 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:02:29.024628 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:29.024599 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:02:29.025105 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:29.025038 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:02:29.025692 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:29.025669 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:02:29.427606 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:29.427570 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:30.885479 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:30.885439 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:31.216457 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.216418 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" podUID="28642a96-d2b2-40e5-b32e-7fa96f97efe9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:02:31.610665 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.610584 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7"] Oct 29 23:02:31.611502 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611481 5610 cpu_manager.go:414] "containerMap: removing container" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" Oct 29 23:02:31.611555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611503 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" Oct 29 23:02:31.611555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611525 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f91681c3-825b-4be3-a29a-6231db8b969a" containerName="maas-api" Oct 29 23:02:31.611555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611530 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91681c3-825b-4be3-a29a-6231db8b969a" containerName="maas-api" Oct 29 23:02:31.611555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611547 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9aead210-f299-4300-9978-f24becb36fed" containerName="payload-processing" Oct 29 23:02:31.611555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611552 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aead210-f299-4300-9978-f24becb36fed" containerName="payload-processing" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611569 5610 cpu_manager.go:414] "containerMap: removing container" podUID="bd36ddb4-7658-4597-a881-a9c36e7fb289" containerName="payload-processing" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611574 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd36ddb4-7658-4597-a881-a9c36e7fb289" containerName="payload-processing" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611592 5610 cpu_manager.go:414] "containerMap: removing container" podUID="442517be-d8d2-4a32-b0c4-255bb0934adc" containerName="payload-pre-processing" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611597 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442517be-d8d2-4a32-b0c4-255bb0934adc" containerName="payload-pre-processing" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611695 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611703 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd36ddb4-7658-4597-a881-a9c36e7fb289" containerName="payload-processing" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611712 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="442517be-d8d2-4a32-b0c4-255bb0934adc" containerName="payload-pre-processing" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611724 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91681c3-825b-4be3-a29a-6231db8b969a" containerName="maas-api" Oct 29 23:02:31.611730 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.611732 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aead210-f299-4300-9978-f24becb36fed" containerName="payload-processing" Oct 29 23:02:31.615830 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.615812 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.638586 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.638547 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7"] Oct 29 23:02:31.644426 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.644385 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1dabb8-openshift-default-dockercfg-skwhs\"" Oct 29 23:02:31.710544 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.710512 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m97k\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-kube-api-access-4m97k\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.710717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.710567 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-data\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.710717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.710664 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-certs\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.710717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.710716 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-credential-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.710829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.710786 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-envoy\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.710865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.710826 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-podinfo\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.710865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.710854 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2d1917-51e2-4f63-ae25-c450d0be21df-istiod-ca-cert\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.710943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.710891 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.711025 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.711001 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-token\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.811868 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.811836 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m97k\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-kube-api-access-4m97k\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812054 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.811903 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-data\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812054 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.811995 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-certs\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812135 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812087 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-credential-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812208 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812192 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-envoy\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812269 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812238 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-podinfo\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812321 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812285 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2d1917-51e2-4f63-ae25-c450d0be21df-istiod-ca-cert\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812401 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812364 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-data\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812401 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812352 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812401 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812381 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-certs\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812774 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812474 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-token\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812774 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812481 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-credential-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.812774 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.812662 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-socket\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.813133 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.813104 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2d1917-51e2-4f63-ae25-c450d0be21df-istiod-ca-cert\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.814571 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.814548 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-envoy\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.815001 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.814983 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-podinfo\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.831577 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.831553 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-token\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.831683 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.831576 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m97k\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-kube-api-access-4m97k\") pod \"e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:31.929656 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:31.929579 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:32.254342 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:02:32.254311 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2d1917_51e2_4f63_ae25_c450d0be21df.slice/crio-080030121e2ef10f93757e6b348e8342a55d5c6d8fa971f27d838f0ac67ef3bc WatchSource:0}: Error finding container 080030121e2ef10f93757e6b348e8342a55d5c6d8fa971f27d838f0ac67ef3bc: Status 404 returned error can't find the container with id 080030121e2ef10f93757e6b348e8342a55d5c6d8fa971f27d838f0ac67ef3bc Oct 29 23:02:32.254995 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:32.254938 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7"] Oct 29 23:02:32.257525 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:32.257482 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:32.257649 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:32.257565 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:32.257649 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:32.257607 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:02:32.283713 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:32.283687 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" event={"ID":"9e2d1917-51e2-4f63-ae25-c450d0be21df","Type":"ContainerStarted","Data":"080030121e2ef10f93757e6b348e8342a55d5c6d8fa971f27d838f0ac67ef3bc"} Oct 29 23:02:32.567240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:32.567074 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:33.322240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.322195 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" event={"ID":"9e2d1917-51e2-4f63-ae25-c450d0be21df","Type":"ContainerStarted","Data":"7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d"} Oct 29 23:02:33.323944 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.323919 5610 generic.go:356] "Generic (PLEG): container finished" podID="28642a96-d2b2-40e5-b32e-7fa96f97efe9" containerID="263cfeb7a14bd60d79aefbb844c06c99407788dbc87a39f41074abe7050f6d4a" exitCode=0 Oct 29 23:02:33.324076 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.324019 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" event={"ID":"28642a96-d2b2-40e5-b32e-7fa96f97efe9","Type":"ContainerDied","Data":"263cfeb7a14bd60d79aefbb844c06c99407788dbc87a39f41074abe7050f6d4a"} Oct 29 23:02:33.354128 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.354106 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 23:02:33.390461 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.390303 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" podStartSLOduration=2.390283749 podStartE2EDuration="2.390283749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:33.39007836 +0000 UTC m=+2815.871395898" watchObservedRunningTime="2026-10-29 23:02:33.390283749 +0000 UTC m=+2815.871601244" Oct 29 23:02:33.435174 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.435135 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28642a96-d2b2-40e5-b32e-7fa96f97efe9-kube-api-access-v9vt8\" (UniqueName: \"kubernetes.io/projected/28642a96-d2b2-40e5-b32e-7fa96f97efe9-kube-api-access-v9vt8\") pod \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\" (UID: \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\") " Oct 29 23:02:33.435372 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.435282 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28642a96-d2b2-40e5-b32e-7fa96f97efe9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28642a96-d2b2-40e5-b32e-7fa96f97efe9-plugins-config-volume\") pod \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\" (UID: \"28642a96-d2b2-40e5-b32e-7fa96f97efe9\") " Oct 29 23:02:33.436601 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.436569 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28642a96-d2b2-40e5-b32e-7fa96f97efe9-plugins-config-volume" pod "28642a96-d2b2-40e5-b32e-7fa96f97efe9" (UID: "28642a96-d2b2-40e5-b32e-7fa96f97efe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:33.445605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.445574 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28642a96-d2b2-40e5-b32e-7fa96f97efe9-kube-api-access-v9vt8" pod "28642a96-d2b2-40e5-b32e-7fa96f97efe9" (UID: "28642a96-d2b2-40e5-b32e-7fa96f97efe9"). InnerVolumeSpecName "kube-api-access-v9vt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:33.538062 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.538019 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9vt8\" (UniqueName: \"kubernetes.io/projected/28642a96-d2b2-40e5-b32e-7fa96f97efe9-kube-api-access-v9vt8\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:33.538062 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.538063 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28642a96-d2b2-40e5-b32e-7fa96f97efe9-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:33.930675 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.930589 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:33.932085 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.932058 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:02:33.932235 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:33.932107 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:02:34.332690 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:34.332657 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" Oct 29 23:02:34.333134 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:34.332696 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z" event={"ID":"28642a96-d2b2-40e5-b32e-7fa96f97efe9","Type":"ContainerDied","Data":"e27e84b22aca67b9a37d7b52e0f63a7fdf64bac2440d991942dc628a248ad8e9"} Oct 29 23:02:34.333134 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:34.332760 5610 scope.go:122] "RemoveContainer" containerID="263cfeb7a14bd60d79aefbb844c06c99407788dbc87a39f41074abe7050f6d4a" Oct 29 23:02:34.406822 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:34.406782 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z"] Oct 29 23:02:34.418356 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:34.418239 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-84304f-6b8c5bd7b-6hb7z"] Oct 29 23:02:34.571414 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:34.567016 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:34.930386 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:34.930350 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:02:34.930596 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:34.930431 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:02:35.637089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:35.637055 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28642a96-d2b2-40e5-b32e-7fa96f97efe9" path="/var/lib/kubelet/pods/28642a96-d2b2-40e5-b32e-7fa96f97efe9/volumes" Oct 29 23:02:35.929974 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:35.929884 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:02:35.929974 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:35.929952 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:02:36.116252 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:36.114780 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:36.286415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:36.285875 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:36.451493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:36.451457 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:36.914882 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:36.914843 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:36.930232 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:36.930200 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:02:36.930441 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:36.930260 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:02:37.075900 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:37.075867 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:37.215766 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:37.215732 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:37.549190 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:37.549061 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:37.930652 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:37.930565 5610 patch_prober.go:29] interesting pod/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 29 23:02:37.930652 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:37.930614 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 29 23:02:38.776128 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:38.776077 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:39.057814 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:39.057732 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:39.058801 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:39.058770 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:39.059704 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:39.059680 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:02:40.238693 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:40.238651 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:40.920895 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:40.920857 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:41.714089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:41.714045 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:43.183415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:43.182674 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:43.715553 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:02:43.715509 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:02:44.288594 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:44.288559 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:44.438290 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:44.438245 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:44.570522 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:44.570435 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r"] Oct 29 23:02:44.570729 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:44.570705 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" containerID="cri-o://b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65" gracePeriod=30 Oct 29 23:02:44.812133 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:44.812101 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:45.129714 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:45.129680 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:45.354439 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:45.354401 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:47.074850 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:47.074796 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:48.026407 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.026352 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:02:48.026609 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.026444 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:02:48.118549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.118514 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:48.261882 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.261845 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:48.501419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.497455 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7"] Oct 29 23:02:48.501419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.498562 5610 cpu_manager.go:414] "containerMap: removing container" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" Oct 29 23:02:48.501419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.498584 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" Oct 29 23:02:48.501419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.498633 5610 cpu_manager.go:414] "containerMap: removing container" podUID="28642a96-d2b2-40e5-b32e-7fa96f97efe9" containerName="payload-processing" Oct 29 23:02:48.501419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.498640 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28642a96-d2b2-40e5-b32e-7fa96f97efe9" containerName="payload-processing" Oct 29 23:02:48.501419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.498834 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="28642a96-d2b2-40e5-b32e-7fa96f97efe9" containerName="payload-processing" Oct 29 23:02:48.501419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.498848 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="89be13e4-0548-49e6-8c81-006939d66752" containerName="istio-proxy" Oct 29 23:02:48.512386 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.512354 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:48.517896 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.516256 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7"] Oct 29 23:02:48.534777 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.534735 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d286f\"" Oct 29 23:02:48.546483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.546450 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d286f-dockercfg-nt6h6\"" Oct 29 23:02:48.644362 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.644326 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc"] Oct 29 23:02:48.647828 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.647808 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:48.674935 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.674881 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc"] Oct 29 23:02:48.684517 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.684484 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da553dc-3f75-4297-97c4-0ca75b6379c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7\" (UID: \"5da553dc-3f75-4297-97c4-0ca75b6379c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:48.684842 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.684820 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkkw\" (UniqueName: \"kubernetes.io/projected/5da553dc-3f75-4297-97c4-0ca75b6379c9-kube-api-access-4mkkw\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7\" (UID: \"5da553dc-3f75-4297-97c4-0ca75b6379c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:48.786487 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.786378 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1f6332-9b86-44ba-84f7-21c12c669084-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc\" (UID: \"0c1f6332-9b86-44ba-84f7-21c12c669084\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:48.786671 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.786536 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcnz\" (UniqueName: \"kubernetes.io/projected/0c1f6332-9b86-44ba-84f7-21c12c669084-kube-api-access-rdcnz\") pod \"payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc\" (UID: \"0c1f6332-9b86-44ba-84f7-21c12c669084\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:48.786671 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.786660 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da553dc-3f75-4297-97c4-0ca75b6379c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7\" (UID: \"5da553dc-3f75-4297-97c4-0ca75b6379c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:48.786807 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.786724 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkkw\" (UniqueName: \"kubernetes.io/projected/5da553dc-3f75-4297-97c4-0ca75b6379c9-kube-api-access-4mkkw\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7\" (UID: \"5da553dc-3f75-4297-97c4-0ca75b6379c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:48.788459 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.788428 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da553dc-3f75-4297-97c4-0ca75b6379c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7\" (UID: \"5da553dc-3f75-4297-97c4-0ca75b6379c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:48.807336 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.807304 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkkw\" (UniqueName: \"kubernetes.io/projected/5da553dc-3f75-4297-97c4-0ca75b6379c9-kube-api-access-4mkkw\") pod \"payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7\" (UID: \"5da553dc-3f75-4297-97c4-0ca75b6379c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:48.855474 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.855446 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:48.889439 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.888168 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcnz\" (UniqueName: \"kubernetes.io/projected/0c1f6332-9b86-44ba-84f7-21c12c669084-kube-api-access-rdcnz\") pod \"payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc\" (UID: \"0c1f6332-9b86-44ba-84f7-21c12c669084\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:48.889439 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.888541 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1f6332-9b86-44ba-84f7-21c12c669084-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc\" (UID: \"0c1f6332-9b86-44ba-84f7-21c12c669084\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:48.889439 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.889372 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1f6332-9b86-44ba-84f7-21c12c669084-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc\" (UID: \"0c1f6332-9b86-44ba-84f7-21c12c669084\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:48.908900 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.908866 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcnz\" (UniqueName: \"kubernetes.io/projected/0c1f6332-9b86-44ba-84f7-21c12c669084-kube-api-access-rdcnz\") pod \"payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc\" (UID: \"0c1f6332-9b86-44ba-84f7-21c12c669084\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:48.985778 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:48.985521 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:49.173237 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:02:49.173198 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da553dc_3f75_4297_97c4_0ca75b6379c9.slice/crio-0c50dacccb0cc653a61ce56525345a4c3fd50e04a5d3f48a8348942eb9b63f08 WatchSource:0}: Error finding container 0c50dacccb0cc653a61ce56525345a4c3fd50e04a5d3f48a8348942eb9b63f08: Status 404 returned error can't find the container with id 0c50dacccb0cc653a61ce56525345a4c3fd50e04a5d3f48a8348942eb9b63f08 Oct 29 23:02:49.173757 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.173726 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7"] Oct 29 23:02:49.298502 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.298376 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc"] Oct 29 23:02:49.299247 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:02:49.299192 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1f6332_9b86_44ba_84f7_21c12c669084.slice/crio-df8013e306bcf6859b8db39fc5ed5885dc2c6c354af8e59456d5383449324603 WatchSource:0}: Error finding container df8013e306bcf6859b8db39fc5ed5885dc2c6c354af8e59456d5383449324603: Status 404 returned error can't find the container with id df8013e306bcf6859b8db39fc5ed5885dc2c6c354af8e59456d5383449324603 Oct 29 23:02:49.378296 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.373078 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:49.551544 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.551447 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" event={"ID":"5da553dc-3f75-4297-97c4-0ca75b6379c9","Type":"ContainerStarted","Data":"759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e"} Oct 29 23:02:49.551740 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.551619 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:02:49.551740 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.551700 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" event={"ID":"5da553dc-3f75-4297-97c4-0ca75b6379c9","Type":"ContainerStarted","Data":"0c50dacccb0cc653a61ce56525345a4c3fd50e04a5d3f48a8348942eb9b63f08"} Oct 29 23:02:49.553349 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.553318 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" event={"ID":"0c1f6332-9b86-44ba-84f7-21c12c669084","Type":"ContainerStarted","Data":"43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406"} Oct 29 23:02:49.553508 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.553363 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" event={"ID":"0c1f6332-9b86-44ba-84f7-21c12c669084","Type":"ContainerStarted","Data":"df8013e306bcf6859b8db39fc5ed5885dc2c6c354af8e59456d5383449324603"} Oct 29 23:02:49.553508 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.553436 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:02:49.594024 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.593000 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" podStartSLOduration=1.592981231 podStartE2EDuration="1.592981231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:49.589630699 +0000 UTC m=+2832.070948197" watchObservedRunningTime="2026-10-29 23:02:49.592981231 +0000 UTC m=+2832.074298728" Oct 29 23:02:49.632697 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.632640 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" podStartSLOduration=1.63262202 podStartE2EDuration="1.63262202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:49.630664558 +0000 UTC m=+2832.111982052" watchObservedRunningTime="2026-10-29 23:02:49.63262202 +0000 UTC m=+2832.113939519" Oct 29 23:02:49.932704 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.932676 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 23:02:49.936370 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936351 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-socket\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.936478 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936415 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-data\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.936478 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936458 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-podinfo\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.936592 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936515 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13e838a-a7be-4b74-8918-5549a1681ba9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13e838a-a7be-4b74-8918-5549a1681ba9-istiod-ca-cert\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.936592 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936544 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-kube-api-access-n9wv5\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-kube-api-access-n9wv5\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.936592 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936570 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-envoy\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.936749 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936602 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-socket" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:49.936749 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936640 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-token\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-token\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.936749 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936684 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-data" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:49.936909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936783 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-credential-socket\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.936966 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.936914 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-certs\") pod \"c13e838a-a7be-4b74-8918-5549a1681ba9\" (UID: \"c13e838a-a7be-4b74-8918-5549a1681ba9\") " Oct 29 23:02:49.937063 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.937041 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-credential-socket" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:49.937310 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.937287 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-certs" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:49.937498 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.937478 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13e838a-a7be-4b74-8918-5549a1681ba9-istiod-ca-cert" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:02:49.937710 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.937690 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.937761 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.937714 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.937761 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.937725 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.937761 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.937735 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.937761 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.937743 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13e838a-a7be-4b74-8918-5549a1681ba9-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:49.941435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.939337 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-kube-api-access-n9wv5" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "kube-api-access-n9wv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:49.941435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.939470 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-podinfo" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:02:49.941435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.939906 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-token" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:02:49.942234 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:49.942212 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-envoy" pod "c13e838a-a7be-4b74-8918-5549a1681ba9" (UID: "c13e838a-a7be-4b74-8918-5549a1681ba9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:02:50.039806 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.039767 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8"] Oct 29 23:02:50.039806 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.039791 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.039806 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.039813 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9wv5\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-kube-api-access-n9wv5\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.040122 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.039827 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.040122 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.039839 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13e838a-a7be-4b74-8918-5549a1681ba9-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:02:50.057467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.054075 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb"] Oct 29 23:02:50.242471 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.242437 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:50.564454 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.564363 5610 generic.go:356] "Generic (PLEG): container finished" podID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerID="b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65" exitCode=0 Oct 29 23:02:50.564454 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.564428 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" Oct 29 23:02:50.564691 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.564458 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" event={"ID":"c13e838a-a7be-4b74-8918-5549a1681ba9","Type":"ContainerDied","Data":"b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65"} Oct 29 23:02:50.564691 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.564514 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r" event={"ID":"c13e838a-a7be-4b74-8918-5549a1681ba9","Type":"ContainerDied","Data":"519fe5e712846e01f06b53364549810d1323a446dadf974ead9cab028335c1bf"} Oct 29 23:02:50.564691 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.564544 5610 scope.go:122] "RemoveContainer" containerID="b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65" Oct 29 23:02:50.591039 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.591012 5610 scope.go:122] "RemoveContainer" containerID="b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65" Oct 29 23:02:50.591370 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:02:50.591332 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65\": container with ID starting with b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65 not found: ID does not exist" containerID="b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65" Oct 29 23:02:50.591511 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.591380 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65"} err="failed to get container status \"b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65\": rpc error: code = NotFound desc = could not find container \"b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65\": container with ID starting with b0a1c2374e70cc5d7e4da89524dce27e12d0070a3a00cff20fa8b9a1ea9dec65 not found: ID does not exist" Oct 29 23:02:50.636687 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.636639 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r"] Oct 29 23:02:50.648544 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:50.648511 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-84304f-openshift-default-6f866bd9c5-p529r"] Oct 29 23:02:51.430468 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:51.430430 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:51.638298 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:51.638264 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" path="/var/lib/kubelet/pods/c13e838a-a7be-4b74-8918-5549a1681ba9/volumes" Oct 29 23:02:53.316107 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:53.316071 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:54.662419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:54.661734 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:55.157893 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:55.157858 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:55.720235 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:55.720199 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq"] Oct 29 23:02:55.879748 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:55.879718 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:56.035708 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:56.035607 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:56.236725 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:56.236691 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:56.518316 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:56.518281 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:56.657865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:56.657827 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:57.370229 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:57.370199 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:57.669738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:57.669647 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:57.826415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:57.825436 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789"] Oct 29 23:02:57.830413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:57.826632 5610 cpu_manager.go:414] "containerMap: removing container" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" Oct 29 23:02:57.830413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:57.826658 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" Oct 29 23:02:57.830413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:57.826941 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13e838a-a7be-4b74-8918-5549a1681ba9" containerName="istio-proxy" Oct 29 23:02:58.039156 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.038916 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789"] Oct 29 23:02:58.041898 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.041873 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:58.068652 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.068622 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l"] Oct 29 23:02:58.071692 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.071658 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9ee1fc\"" Oct 29 23:02:58.090305 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.090241 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l"] Oct 29 23:02:58.090499 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.090364 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:58.110766 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.110737 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9ee1fc-dockercfg-lk52g\"" Oct 29 23:02:58.143579 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.143555 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjmg2\" (UniqueName: \"kubernetes.io/projected/ce072c8e-2cca-4616-916d-2d2ddb52d113-kube-api-access-gjmg2\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789\" (UID: \"ce072c8e-2cca-4616-916d-2d2ddb52d113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:58.143748 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.143733 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56496049-8920-4b5f-b18d-7d536f38feb0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l\" (UID: \"56496049-8920-4b5f-b18d-7d536f38feb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:58.143800 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.143781 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td25g\" (UniqueName: \"kubernetes.io/projected/56496049-8920-4b5f-b18d-7d536f38feb0-kube-api-access-td25g\") pod \"payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l\" (UID: \"56496049-8920-4b5f-b18d-7d536f38feb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:58.143893 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.143880 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce072c8e-2cca-4616-916d-2d2ddb52d113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789\" (UID: \"ce072c8e-2cca-4616-916d-2d2ddb52d113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:58.245251 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.245210 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce072c8e-2cca-4616-916d-2d2ddb52d113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789\" (UID: \"ce072c8e-2cca-4616-916d-2d2ddb52d113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:58.245472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.245281 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjmg2\" (UniqueName: \"kubernetes.io/projected/ce072c8e-2cca-4616-916d-2d2ddb52d113-kube-api-access-gjmg2\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789\" (UID: \"ce072c8e-2cca-4616-916d-2d2ddb52d113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:58.245570 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.245514 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56496049-8920-4b5f-b18d-7d536f38feb0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l\" (UID: \"56496049-8920-4b5f-b18d-7d536f38feb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:58.245653 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.245590 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td25g\" (UniqueName: \"kubernetes.io/projected/56496049-8920-4b5f-b18d-7d536f38feb0-kube-api-access-td25g\") pod \"payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l\" (UID: \"56496049-8920-4b5f-b18d-7d536f38feb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:58.246145 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.246118 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce072c8e-2cca-4616-916d-2d2ddb52d113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789\" (UID: \"ce072c8e-2cca-4616-916d-2d2ddb52d113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:58.246290 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.246275 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56496049-8920-4b5f-b18d-7d536f38feb0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l\" (UID: \"56496049-8920-4b5f-b18d-7d536f38feb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:58.316458 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.307978 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjmg2\" (UniqueName: \"kubernetes.io/projected/ce072c8e-2cca-4616-916d-2d2ddb52d113-kube-api-access-gjmg2\") pod \"payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789\" (UID: \"ce072c8e-2cca-4616-916d-2d2ddb52d113\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:58.333414 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.330318 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td25g\" (UniqueName: \"kubernetes.io/projected/56496049-8920-4b5f-b18d-7d536f38feb0-kube-api-access-td25g\") pod \"payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l\" (UID: \"56496049-8920-4b5f-b18d-7d536f38feb0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:58.413774 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.413744 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:58.426211 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.426187 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:58.522444 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.515886 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:58.751233 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:02:58.751199 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce072c8e_2cca_4616_916d_2d2ddb52d113.slice/crio-fe27c4b85d49da6f90397e51f93727f077ec8a179fe2a633e84b95a9a121aad8 WatchSource:0}: Error finding container fe27c4b85d49da6f90397e51f93727f077ec8a179fe2a633e84b95a9a121aad8: Status 404 returned error can't find the container with id fe27c4b85d49da6f90397e51f93727f077ec8a179fe2a633e84b95a9a121aad8 Oct 29 23:02:58.754422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.754370 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789"] Oct 29 23:02:58.758089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.758061 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:02:58.781245 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:58.781215 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l"] Oct 29 23:02:58.789960 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:02:58.789926 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56496049_8920_4b5f_b18d_7d536f38feb0.slice/crio-69067fa4a662347c7dc655e658cd98cd4fb2b2538227cc900ff5b291eeba5995 WatchSource:0}: Error finding container 69067fa4a662347c7dc655e658cd98cd4fb2b2538227cc900ff5b291eeba5995: Status 404 returned error can't find the container with id 69067fa4a662347c7dc655e658cd98cd4fb2b2538227cc900ff5b291eeba5995 Oct 29 23:02:59.645425 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:59.645354 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" event={"ID":"56496049-8920-4b5f-b18d-7d536f38feb0","Type":"ContainerStarted","Data":"d246eb1f7eefb4c54cb49940f044ceef2ab0ca40b616384a1ecc6d1ae6b65f95"} Oct 29 23:02:59.645933 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:59.645456 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" event={"ID":"56496049-8920-4b5f-b18d-7d536f38feb0","Type":"ContainerStarted","Data":"69067fa4a662347c7dc655e658cd98cd4fb2b2538227cc900ff5b291eeba5995"} Oct 29 23:02:59.645933 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:59.645679 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:02:59.647211 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:59.647183 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" event={"ID":"ce072c8e-2cca-4616-916d-2d2ddb52d113","Type":"ContainerStarted","Data":"1d789adba7079dd9d42cea1e90f4de278b43b35be115d5c6e70e5ca20ed60325"} Oct 29 23:02:59.647324 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:59.647231 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" event={"ID":"ce072c8e-2cca-4616-916d-2d2ddb52d113","Type":"ContainerStarted","Data":"fe27c4b85d49da6f90397e51f93727f077ec8a179fe2a633e84b95a9a121aad8"} Oct 29 23:02:59.647324 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:59.647253 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:02:59.685159 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:59.685094 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" podStartSLOduration=2.6850739470000002 podStartE2EDuration="2.685073947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:59.682990097 +0000 UTC m=+2842.164307592" watchObservedRunningTime="2026-10-29 23:02:59.685073947 +0000 UTC m=+2842.166391445" Oct 29 23:02:59.722492 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:02:59.722435 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" podStartSLOduration=2.722423207 podStartE2EDuration="2.722423207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:02:59.721363027 +0000 UTC m=+2842.202680525" watchObservedRunningTime="2026-10-29 23:02:59.722423207 +0000 UTC m=+2842.203740697" Oct 29 23:03:00.074445 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:00.074371 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:03:00.171539 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:00.168155 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:00.277374 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:00.277344 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:03:00.461449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:00.461422 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f_91a0bb3e-f76d-4030-a53c-ac5905587c93/payload-processing/0.log" Oct 29 23:03:00.568089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:00.568059 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:03:00.568510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:00.568481 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:03:00.655908 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:00.655875 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f_91a0bb3e-f76d-4030-a53c-ac5905587c93/payload-processing/0.log" Oct 29 23:03:00.721129 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:00.721048 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" podUID="b8ad7ba1-789d-4681-a532-39068330f54f" containerName="payload-pre-processing" containerID="cri-o://1ef210596f270b6d091d2af346aae69be2b645708b3ebf5a4bcfeb010a6fabff" gracePeriod=25 Oct 29 23:03:02.000180 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:02.000147 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:03.646952 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:03.646888 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:04.472814 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:04.472781 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:04.639837 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:04.639806 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:05.052358 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.052309 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:05.072843 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.072803 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" containerID="cri-o://7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f" gracePeriod=45 Oct 29 23:03:05.096931 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.096883 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" containerID="cri-o://7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2" gracePeriod=45 Oct 29 23:03:05.436021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.435996 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 23:03:05.439872 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.439835 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 23:03:05.498984 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.498954 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7424bca-507a-4c13-a526-a67de510814e-kube-api-access-spm7v\" (UniqueName: \"kubernetes.io/projected/b7424bca-507a-4c13-a526-a67de510814e-kube-api-access-spm7v\") pod \"b7424bca-507a-4c13-a526-a67de510814e\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " Oct 29 23:03:05.499265 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.499244 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5def10-271f-4669-896d-9183f5015de9-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5def10-271f-4669-896d-9183f5015de9-tls-certs\") pod \"cf5def10-271f-4669-896d-9183f5015de9\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " Oct 29 23:03:05.499548 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.499515 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-dshm\") pod \"b7424bca-507a-4c13-a526-a67de510814e\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " Oct 29 23:03:05.501386 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.501363 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-home\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-home\") pod \"cf5def10-271f-4669-896d-9183f5015de9\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " Oct 29 23:03:05.501650 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.501634 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-tmp-dir\") pod \"cf5def10-271f-4669-896d-9183f5015de9\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " Oct 29 23:03:05.501795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.501780 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-model-cache\") pod \"b7424bca-507a-4c13-a526-a67de510814e\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " Oct 29 23:03:05.501996 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.501980 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-dshm\") pod \"cf5def10-271f-4669-896d-9183f5015de9\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " Oct 29 23:03:05.502462 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.502425 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-model-cache" pod "b7424bca-507a-4c13-a526-a67de510814e" (UID: "b7424bca-507a-4c13-a526-a67de510814e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.502581 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.502500 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-home" pod "cf5def10-271f-4669-896d-9183f5015de9" (UID: "cf5def10-271f-4669-896d-9183f5015de9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.502581 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.502546 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-tmp-dir" pod "cf5def10-271f-4669-896d-9183f5015de9" (UID: "cf5def10-271f-4669-896d-9183f5015de9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.502735 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.502713 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5def10-271f-4669-896d-9183f5015de9-kube-api-access-ml8th\" (UniqueName: \"kubernetes.io/projected/cf5def10-271f-4669-896d-9183f5015de9-kube-api-access-ml8th\") pod \"cf5def10-271f-4669-896d-9183f5015de9\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " Oct 29 23:03:05.502938 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.502915 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7424bca-507a-4c13-a526-a67de510814e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7424bca-507a-4c13-a526-a67de510814e-tls-certs\") pod \"b7424bca-507a-4c13-a526-a67de510814e\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " Oct 29 23:03:05.503123 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.503109 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-home\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-home\") pod \"b7424bca-507a-4c13-a526-a67de510814e\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " Oct 29 23:03:05.503260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.503247 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-tmp-dir\") pod \"b7424bca-507a-4c13-a526-a67de510814e\" (UID: \"b7424bca-507a-4c13-a526-a67de510814e\") " Oct 29 23:03:05.503495 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.503478 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-model-cache\") pod \"cf5def10-271f-4669-896d-9183f5015de9\" (UID: \"cf5def10-271f-4669-896d-9183f5015de9\") " Oct 29 23:03:05.503937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.503901 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-tmp-dir" pod "b7424bca-507a-4c13-a526-a67de510814e" (UID: "b7424bca-507a-4c13-a526-a67de510814e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.504875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.504136 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-model-cache" pod "cf5def10-271f-4669-896d-9183f5015de9" (UID: "cf5def10-271f-4669-896d-9183f5015de9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.504875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.504698 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.504875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.504721 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.504875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.504735 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.504875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.504747 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.504875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.504760 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.504875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.504834 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-home" pod "b7424bca-507a-4c13-a526-a67de510814e" (UID: "b7424bca-507a-4c13-a526-a67de510814e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.505790 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.505747 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-dshm" pod "b7424bca-507a-4c13-a526-a67de510814e" (UID: "b7424bca-507a-4c13-a526-a67de510814e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.511817 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.510556 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7424bca-507a-4c13-a526-a67de510814e-kube-api-access-spm7v" pod "b7424bca-507a-4c13-a526-a67de510814e" (UID: "b7424bca-507a-4c13-a526-a67de510814e"). InnerVolumeSpecName "kube-api-access-spm7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:05.511817 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.510705 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5def10-271f-4669-896d-9183f5015de9-tls-certs" pod "cf5def10-271f-4669-896d-9183f5015de9" (UID: "cf5def10-271f-4669-896d-9183f5015de9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:05.511817 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.510895 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5def10-271f-4669-896d-9183f5015de9-kube-api-access-ml8th" pod "cf5def10-271f-4669-896d-9183f5015de9" (UID: "cf5def10-271f-4669-896d-9183f5015de9"). InnerVolumeSpecName "kube-api-access-ml8th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:05.520171 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.520114 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7424bca-507a-4c13-a526-a67de510814e-tls-certs" pod "b7424bca-507a-4c13-a526-a67de510814e" (UID: "b7424bca-507a-4c13-a526-a67de510814e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:05.520455 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.520429 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-dshm" pod "cf5def10-271f-4669-896d-9183f5015de9" (UID: "cf5def10-271f-4669-896d-9183f5015de9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:05.606757 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.606680 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spm7v\" (UniqueName: \"kubernetes.io/projected/b7424bca-507a-4c13-a526-a67de510814e-kube-api-access-spm7v\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.606757 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.606718 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5def10-271f-4669-896d-9183f5015de9-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.606757 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.606733 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.606757 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.606745 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5def10-271f-4669-896d-9183f5015de9-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.606757 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.606758 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8th\" (UniqueName: \"kubernetes.io/projected/cf5def10-271f-4669-896d-9183f5015de9-kube-api-access-ml8th\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.607123 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.606771 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7424bca-507a-4c13-a526-a67de510814e-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.607123 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.606784 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7424bca-507a-4c13-a526-a67de510814e-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:05.710733 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.710690 5610 generic.go:356] "Generic (PLEG): container finished" podID="cf5def10-271f-4669-896d-9183f5015de9" containerID="7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2" exitCode=0 Oct 29 23:03:05.710928 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.710756 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" Oct 29 23:03:05.710928 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.710770 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" event={"ID":"cf5def10-271f-4669-896d-9183f5015de9","Type":"ContainerDied","Data":"7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2"} Oct 29 23:03:05.710928 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.710816 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb" event={"ID":"cf5def10-271f-4669-896d-9183f5015de9","Type":"ContainerDied","Data":"47d18f0f83da8c0db8eca5dc8252627d657061d3fa6b7baff3fc42ffc4a22629"} Oct 29 23:03:05.710928 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.710837 5610 scope.go:122] "RemoveContainer" containerID="7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2" Oct 29 23:03:05.712945 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.712917 5610 generic.go:356] "Generic (PLEG): container finished" podID="b7424bca-507a-4c13-a526-a67de510814e" containerID="7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f" exitCode=0 Oct 29 23:03:05.713053 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.713007 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" event={"ID":"b7424bca-507a-4c13-a526-a67de510814e","Type":"ContainerDied","Data":"7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f"} Oct 29 23:03:05.713053 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.713031 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" event={"ID":"b7424bca-507a-4c13-a526-a67de510814e","Type":"ContainerDied","Data":"6c0ad7690225e22000ff8f061d4e6a8f72af25a02f9e8c78cf3d6bc9edd89183"} Oct 29 23:03:05.713194 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.713054 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" Oct 29 23:03:05.723409 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.723379 5610 scope.go:122] "RemoveContainer" containerID="7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2" Oct 29 23:03:05.724449 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:05.723704 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2\": container with ID starting with 7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2 not found: ID does not exist" containerID="7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2" Oct 29 23:03:05.724449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.723741 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2"} err="failed to get container status \"7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2\": rpc error: code = NotFound desc = could not find container \"7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2\": container with ID starting with 7ff95abfe5029da5ef73b2eaa17d9a7cd94bf22a9c184b6c8d572c1430d75cd2 not found: ID does not exist" Oct 29 23:03:05.724449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.723760 5610 scope.go:122] "RemoveContainer" containerID="7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f" Oct 29 23:03:05.736475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.736457 5610 scope.go:122] "RemoveContainer" containerID="7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f" Oct 29 23:03:05.737120 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:05.736768 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f\": container with ID starting with 7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f not found: ID does not exist" containerID="7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f" Oct 29 23:03:05.737120 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.736793 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f"} err="failed to get container status \"7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f\": rpc error: code = NotFound desc = could not find container \"7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f\": container with ID starting with 7daffe8f8ed793b26bab9af4462b78c16b1ef37f9dab0710f1b762a33bacc24f not found: ID does not exist" Oct 29 23:03:05.762987 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.762961 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb"] Oct 29 23:03:05.768989 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.768961 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-w2-8d3632-kserve-5f6856f47-s9szb"] Oct 29 23:03:05.800084 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.799898 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8"] Oct 29 23:03:05.803806 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:05.803778 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8"] Oct 29 23:03:06.357740 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:06.357659 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8d3632/distinct-2-w2-8d3632-kserve-65b5855c6d-2dbf8" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:03:06.669991 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:06.669915 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:06.832387 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:06.832345 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" podUID="b8ad7ba1-789d-4681-a532-39068330f54f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:03:07.637717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:07.637685 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7424bca-507a-4c13-a526-a67de510814e" path="/var/lib/kubelet/pods/b7424bca-507a-4c13-a526-a67de510814e/volumes" Oct 29 23:03:07.638120 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:07.638106 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5def10-271f-4669-896d-9183f5015de9" path="/var/lib/kubelet/pods/cf5def10-271f-4669-896d-9183f5015de9/volumes" Oct 29 23:03:07.785774 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:07.785737 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:07.941503 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:07.941430 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:08.131285 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.131242 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd"] Oct 29 23:03:08.132608 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.132577 5610 cpu_manager.go:414] "containerMap: removing container" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" Oct 29 23:03:08.132608 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.132605 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" Oct 29 23:03:08.132827 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.132636 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" Oct 29 23:03:08.132827 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.132647 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" Oct 29 23:03:08.132827 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.132818 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5def10-271f-4669-896d-9183f5015de9" containerName="main" Oct 29 23:03:08.132982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.132832 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7424bca-507a-4c13-a526-a67de510814e" containerName="main" Oct 29 23:03:08.155448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.155417 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:08.159358 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.159333 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd"] Oct 29 23:03:08.187289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.187228 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3e2b2c\"" Oct 29 23:03:08.187658 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.187296 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3e2b2c-dockercfg-2t6kn\"" Oct 29 23:03:08.252684 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.252652 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhg4q\" (UniqueName: \"kubernetes.io/projected/460af1df-6d95-4d73-923a-61bd9ba7f11a-kube-api-access-bhg4q\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd\" (UID: \"460af1df-6d95-4d73-923a-61bd9ba7f11a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:08.252855 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.252707 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460af1df-6d95-4d73-923a-61bd9ba7f11a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd\" (UID: \"460af1df-6d95-4d73-923a-61bd9ba7f11a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:08.354885 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.354829 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhg4q\" (UniqueName: \"kubernetes.io/projected/460af1df-6d95-4d73-923a-61bd9ba7f11a-kube-api-access-bhg4q\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd\" (UID: \"460af1df-6d95-4d73-923a-61bd9ba7f11a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:08.355067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.354915 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460af1df-6d95-4d73-923a-61bd9ba7f11a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd\" (UID: \"460af1df-6d95-4d73-923a-61bd9ba7f11a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:08.355838 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.355794 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460af1df-6d95-4d73-923a-61bd9ba7f11a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd\" (UID: \"460af1df-6d95-4d73-923a-61bd9ba7f11a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:08.398996 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.380719 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhg4q\" (UniqueName: \"kubernetes.io/projected/460af1df-6d95-4d73-923a-61bd9ba7f11a-kube-api-access-bhg4q\") pod \"payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd\" (UID: \"460af1df-6d95-4d73-923a-61bd9ba7f11a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:08.479139 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.479107 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:08.543218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.541638 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp"] Oct 29 23:03:08.543218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.542233 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" podUID="3940cff7-5a19-4fc8-9c07-9f8233186f2e" containerName="maas-api" containerID="cri-o://a938f95471ae6f9def363c910a62944dca927abdb4735f167c1820dda8801967" gracePeriod=30 Oct 29 23:03:08.646587 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.646533 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:08.784622 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.784583 5610 generic.go:356] "Generic (PLEG): container finished" podID="3940cff7-5a19-4fc8-9c07-9f8233186f2e" containerID="a938f95471ae6f9def363c910a62944dca927abdb4735f167c1820dda8801967" exitCode=0 Oct 29 23:03:08.784777 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.784654 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" event={"ID":"3940cff7-5a19-4fc8-9c07-9f8233186f2e","Type":"ContainerDied","Data":"a938f95471ae6f9def363c910a62944dca927abdb4735f167c1820dda8801967"} Oct 29 23:03:08.861604 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.861527 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd"] Oct 29 23:03:08.940427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:08.940380 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 23:03:09.123701 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.123672 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3940cff7-5a19-4fc8-9c07-9f8233186f2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3940cff7-5a19-4fc8-9c07-9f8233186f2e-openshift-service-ca\") pod \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " Oct 29 23:03:09.124185 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.124142 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3940cff7-5a19-4fc8-9c07-9f8233186f2e-kube-api-access-qb24q\" (UniqueName: \"kubernetes.io/projected/3940cff7-5a19-4fc8-9c07-9f8233186f2e-kube-api-access-qb24q\") pod \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " Oct 29 23:03:09.124331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.124312 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3940cff7-5a19-4fc8-9c07-9f8233186f2e-openshift-service-ca" pod "3940cff7-5a19-4fc8-9c07-9f8233186f2e" (UID: "3940cff7-5a19-4fc8-9c07-9f8233186f2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:09.124689 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.124364 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-metrics-certs\") pod \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " Oct 29 23:03:09.124893 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.124873 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-maas-api-tls\") pod \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\" (UID: \"3940cff7-5a19-4fc8-9c07-9f8233186f2e\") " Oct 29 23:03:09.125693 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.125666 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3940cff7-5a19-4fc8-9c07-9f8233186f2e-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:09.129765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.129677 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-maas-api-tls" pod "3940cff7-5a19-4fc8-9c07-9f8233186f2e" (UID: "3940cff7-5a19-4fc8-9c07-9f8233186f2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:09.129765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.129716 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-metrics-certs" pod "3940cff7-5a19-4fc8-9c07-9f8233186f2e" (UID: "3940cff7-5a19-4fc8-9c07-9f8233186f2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:09.129765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.129728 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3940cff7-5a19-4fc8-9c07-9f8233186f2e-kube-api-access-qb24q" pod "3940cff7-5a19-4fc8-9c07-9f8233186f2e" (UID: "3940cff7-5a19-4fc8-9c07-9f8233186f2e"). InnerVolumeSpecName "kube-api-access-qb24q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:09.189916 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.189883 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:09.226728 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.226697 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:09.226728 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.226724 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb24q\" (UniqueName: \"kubernetes.io/projected/3940cff7-5a19-4fc8-9c07-9f8233186f2e-kube-api-access-qb24q\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:09.226728 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.226733 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3940cff7-5a19-4fc8-9c07-9f8233186f2e-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:09.342099 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.342061 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:09.509422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.506639 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:09.794481 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.794447 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" Oct 29 23:03:09.795140 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.794448 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp" event={"ID":"3940cff7-5a19-4fc8-9c07-9f8233186f2e","Type":"ContainerDied","Data":"c7335d8bc39398b4e7f72873e68e0c9509371b7c2a374ddddf2c188789ac1d05"} Oct 29 23:03:09.795140 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.794620 5610 scope.go:122] "RemoveContainer" containerID="a938f95471ae6f9def363c910a62944dca927abdb4735f167c1820dda8801967" Oct 29 23:03:09.796661 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.796637 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" event={"ID":"460af1df-6d95-4d73-923a-61bd9ba7f11a","Type":"ContainerStarted","Data":"c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a"} Oct 29 23:03:09.796767 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.796692 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" event={"ID":"460af1df-6d95-4d73-923a-61bd9ba7f11a","Type":"ContainerStarted","Data":"d38eb1aea6e2b9fb9ebc918399a2e82a95afd6d7e3097c569a55b634cf2df455"} Oct 29 23:03:09.796767 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.796724 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:09.867430 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.867379 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp"] Oct 29 23:03:09.884728 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.884696 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-879596-56695cd8fc-l5hfp"] Oct 29 23:03:09.937888 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.937785 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" podStartSLOduration=1.937772744 podStartE2EDuration="1.937772744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:09.935372614 +0000 UTC m=+2852.416690107" watchObservedRunningTime="2026-10-29 23:03:09.937772744 +0000 UTC m=+2852.419090236" Oct 29 23:03:09.949040 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:09.949008 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:10.666241 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:10.666210 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:03:10.666889 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:10.666863 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:03:11.639032 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:11.638995 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3940cff7-5a19-4fc8-9c07-9f8233186f2e" path="/var/lib/kubelet/pods/3940cff7-5a19-4fc8-9c07-9f8233186f2e/volumes" Oct 29 23:03:11.755452 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:11.754962 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:13.377322 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:13.377285 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:13.713794 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:13.713761 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:03:14.944193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:14.944159 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:16.117833 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:16.117794 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:16.320415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:16.314901 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:16.504774 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:16.504746 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:16.832883 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:16.832794 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" podUID="b8ad7ba1-789d-4681-a532-39068330f54f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:03:16.927022 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:16.926984 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:17.003904 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:17.003871 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh"] Oct 29 23:03:17.004077 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:17.004057 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" containerID="cri-o://36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401" gracePeriod=30 Oct 29 23:03:17.230134 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:17.230101 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:18.020581 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.020543 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:18.063845 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.063809 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc"] Oct 29 23:03:18.064057 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.064032 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" podUID="7e826947-7906-41b9-9bd5-2147dbb0c076" containerName="payload-processing" containerID="cri-o://4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2" gracePeriod=30 Oct 29 23:03:18.199043 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.194860 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:18.404475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.383249 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:18.514371 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.513972 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:03:18.616374 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.616342 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:18.626678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.626648 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e826947-7906-41b9-9bd5-2147dbb0c076-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e826947-7906-41b9-9bd5-2147dbb0c076-plugins-config-volume\") pod \"7e826947-7906-41b9-9bd5-2147dbb0c076\" (UID: \"7e826947-7906-41b9-9bd5-2147dbb0c076\") " Oct 29 23:03:18.626849 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.626704 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e826947-7906-41b9-9bd5-2147dbb0c076-kube-api-access-qdgzb\" (UniqueName: \"kubernetes.io/projected/7e826947-7906-41b9-9bd5-2147dbb0c076-kube-api-access-qdgzb\") pod \"7e826947-7906-41b9-9bd5-2147dbb0c076\" (UID: \"7e826947-7906-41b9-9bd5-2147dbb0c076\") " Oct 29 23:03:18.627139 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.627111 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e826947-7906-41b9-9bd5-2147dbb0c076-plugins-config-volume" pod "7e826947-7906-41b9-9bd5-2147dbb0c076" (UID: "7e826947-7906-41b9-9bd5-2147dbb0c076"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:18.627304 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.627284 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e826947-7906-41b9-9bd5-2147dbb0c076-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.629210 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.629181 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e826947-7906-41b9-9bd5-2147dbb0c076-kube-api-access-qdgzb" pod "7e826947-7906-41b9-9bd5-2147dbb0c076" (UID: "7e826947-7906-41b9-9bd5-2147dbb0c076"). InnerVolumeSpecName "kube-api-access-qdgzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:18.728242 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.728206 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdgzb\" (UniqueName: \"kubernetes.io/projected/7e826947-7906-41b9-9bd5-2147dbb0c076-kube-api-access-qdgzb\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:18.761367 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.761332 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:18.885363 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.885329 5610 generic.go:356] "Generic (PLEG): container finished" podID="7e826947-7906-41b9-9bd5-2147dbb0c076" containerID="4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2" exitCode=0 Oct 29 23:03:18.885603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.885420 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" Oct 29 23:03:18.885603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.885460 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" event={"ID":"7e826947-7906-41b9-9bd5-2147dbb0c076","Type":"ContainerDied","Data":"4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2"} Oct 29 23:03:18.885603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.885521 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc" event={"ID":"7e826947-7906-41b9-9bd5-2147dbb0c076","Type":"ContainerDied","Data":"c10ef1da355f3f56d6d6f27924ac7974db9ac2473517a616898d8fc5c1608fbe"} Oct 29 23:03:18.885603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.885553 5610 scope.go:122] "RemoveContainer" containerID="4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2" Oct 29 23:03:18.898875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.898858 5610 scope.go:122] "RemoveContainer" containerID="4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2" Oct 29 23:03:18.899149 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:18.899127 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2\": container with ID starting with 4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2 not found: ID does not exist" containerID="4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2" Oct 29 23:03:18.899215 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.899156 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2"} err="failed to get container status \"4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2\": rpc error: code = NotFound desc = could not find container \"4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2\": container with ID starting with 4b409ea2c57a6471dfe55244828b6c4d1b3ce140f904b2351ed728ed6fca63a2 not found: ID does not exist" Oct 29 23:03:18.907230 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.907209 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:18.938491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.938464 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc"] Oct 29 23:03:18.943090 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:18.943068 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-14b0c3-64f964956d-d45gc"] Oct 29 23:03:19.275851 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:19.275819 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:19.645050 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:19.644998 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e826947-7906-41b9-9bd5-2147dbb0c076" path="/var/lib/kubelet/pods/7e826947-7906-41b9-9bd5-2147dbb0c076/volumes" Oct 29 23:03:20.466989 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:20.466733 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:20.633077 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:20.633046 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:20.785796 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:20.785767 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:20.810673 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:20.810644 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:03:20.910249 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:20.910217 5610 generic.go:356] "Generic (PLEG): container finished" podID="b8ad7ba1-789d-4681-a532-39068330f54f" containerID="1ef210596f270b6d091d2af346aae69be2b645708b3ebf5a4bcfeb010a6fabff" exitCode=0 Oct 29 23:03:20.910482 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:20.910344 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" event={"ID":"b8ad7ba1-789d-4681-a532-39068330f54f","Type":"ContainerDied","Data":"1ef210596f270b6d091d2af346aae69be2b645708b3ebf5a4bcfeb010a6fabff"} Oct 29 23:03:21.045744 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.045713 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 23:03:21.185413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.184500 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ad7ba1-789d-4681-a532-39068330f54f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ad7ba1-789d-4681-a532-39068330f54f-plugins-config-volume\") pod \"b8ad7ba1-789d-4681-a532-39068330f54f\" (UID: \"b8ad7ba1-789d-4681-a532-39068330f54f\") " Oct 29 23:03:21.185413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.184795 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ad7ba1-789d-4681-a532-39068330f54f-kube-api-access-fpm8f\" (UniqueName: \"kubernetes.io/projected/b8ad7ba1-789d-4681-a532-39068330f54f-kube-api-access-fpm8f\") pod \"b8ad7ba1-789d-4681-a532-39068330f54f\" (UID: \"b8ad7ba1-789d-4681-a532-39068330f54f\") " Oct 29 23:03:21.185413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.184858 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ad7ba1-789d-4681-a532-39068330f54f-plugins-config-volume" pod "b8ad7ba1-789d-4681-a532-39068330f54f" (UID: "b8ad7ba1-789d-4681-a532-39068330f54f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:21.185413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.185321 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ad7ba1-789d-4681-a532-39068330f54f-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.207478 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.187545 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ad7ba1-789d-4681-a532-39068330f54f-kube-api-access-fpm8f" pod "b8ad7ba1-789d-4681-a532-39068330f54f" (UID: "b8ad7ba1-789d-4681-a532-39068330f54f"). InnerVolumeSpecName "kube-api-access-fpm8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:21.286955 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.286849 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpm8f\" (UniqueName: \"kubernetes.io/projected/b8ad7ba1-789d-4681-a532-39068330f54f-kube-api-access-fpm8f\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:21.918372 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.918285 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" Oct 29 23:03:21.918372 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.918316 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq" event={"ID":"b8ad7ba1-789d-4681-a532-39068330f54f","Type":"ContainerDied","Data":"739a85647d1741bace4206853fc716c5384c897318739ad9f2afa360ef014223"} Oct 29 23:03:21.918910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.918389 5610 scope.go:122] "RemoveContainer" containerID="1ef210596f270b6d091d2af346aae69be2b645708b3ebf5a4bcfeb010a6fabff" Oct 29 23:03:21.969160 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.969129 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq"] Oct 29 23:03:21.980700 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:21.980670 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-8d3632-74d48c769f-8ggxq"] Oct 29 23:03:22.312518 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.306198 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:22.323606 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.323581 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 23:03:22.409197 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409161 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-certs\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.409389 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409268 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-kube-api-access-l2fq4\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-kube-api-access-l2fq4\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.409389 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409319 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-envoy\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.409572 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409428 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-credential-socket\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.409572 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409462 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/573f8be8-460f-4353-b9ee-e85751503f91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573f8be8-460f-4353-b9ee-e85751503f91-istio-podinfo\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.409694 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409559 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-certs" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.409694 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409603 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-socket\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.409694 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409641 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-istio-token\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-istio-token\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.409868 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409703 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573f8be8-460f-4353-b9ee-e85751503f91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573f8be8-460f-4353-b9ee-e85751503f91-istiod-ca-cert\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.409868 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.409795 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-data\") pod \"573f8be8-460f-4353-b9ee-e85751503f91\" (UID: \"573f8be8-460f-4353-b9ee-e85751503f91\") " Oct 29 23:03:22.411106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.410538 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.411106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.411069 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573f8be8-460f-4353-b9ee-e85751503f91-istiod-ca-cert" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:22.411283 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.411135 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-data" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.411597 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.411564 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-credential-socket" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.411811 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.411781 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-socket" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.413933 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.413904 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-envoy" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:03:22.414430 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.414373 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-istio-token" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:22.414536 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.414386 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/573f8be8-460f-4353-b9ee-e85751503f91-istio-podinfo" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:03:22.414967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.414940 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-kube-api-access-l2fq4" pod "573f8be8-460f-4353-b9ee-e85751503f91" (UID: "573f8be8-460f-4353-b9ee-e85751503f91"). InnerVolumeSpecName "kube-api-access-l2fq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:22.469975 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.469938 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:22.512491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.512071 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.512491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.512109 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/573f8be8-460f-4353-b9ee-e85751503f91-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.512491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.512126 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.512491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.512140 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.512491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.512242 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/573f8be8-460f-4353-b9ee-e85751503f91-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.512491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.512258 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.512491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.512271 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2fq4\" (UniqueName: \"kubernetes.io/projected/573f8be8-460f-4353-b9ee-e85751503f91-kube-api-access-l2fq4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.512491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.512285 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/573f8be8-460f-4353-b9ee-e85751503f91-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:22.928872 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.928786 5610 generic.go:356] "Generic (PLEG): container finished" podID="573f8be8-460f-4353-b9ee-e85751503f91" containerID="36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401" exitCode=0 Oct 29 23:03:22.928872 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.928858 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" Oct 29 23:03:22.929449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.928866 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" event={"ID":"573f8be8-460f-4353-b9ee-e85751503f91","Type":"ContainerDied","Data":"36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401"} Oct 29 23:03:22.929449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.928942 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh" event={"ID":"573f8be8-460f-4353-b9ee-e85751503f91","Type":"ContainerDied","Data":"000ac277d8e8fb33b4d9b05bf28a93f319d82789dac27586439b0322f59ca990"} Oct 29 23:03:22.929449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.928966 5610 scope.go:122] "RemoveContainer" containerID="36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401" Oct 29 23:03:22.940370 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.940347 5610 scope.go:122] "RemoveContainer" containerID="0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1" Oct 29 23:03:22.950864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.950847 5610 scope.go:122] "RemoveContainer" containerID="36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401" Oct 29 23:03:22.951164 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:22.951144 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401\": container with ID starting with 36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401 not found: ID does not exist" containerID="36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401" Oct 29 23:03:22.951241 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.951172 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401"} err="failed to get container status \"36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401\": rpc error: code = NotFound desc = could not find container \"36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401\": container with ID starting with 36d74f3f700902415464c936f8593f4c1d79314b611eda290be529a489861401 not found: ID does not exist" Oct 29 23:03:22.951241 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.951195 5610 scope.go:122] "RemoveContainer" containerID="0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1" Oct 29 23:03:22.951474 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:22.951450 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1\": container with ID starting with 0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1 not found: ID does not exist" containerID="0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1" Oct 29 23:03:22.951547 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.951482 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1"} err="failed to get container status \"0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1\": rpc error: code = NotFound desc = could not find container \"0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1\": container with ID starting with 0e5785630a1eb1f81d4ab8a324211dd47ae0ef8fab421403055562d3c260fbc1 not found: ID does not exist" Oct 29 23:03:22.981959 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.981930 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh"] Oct 29 23:03:22.992877 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:22.992850 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-879596-openshift-default-6b7d57dcc5-6npbh"] Oct 29 23:03:23.027419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:23.026379 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:23.640976 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:23.640932 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573f8be8-460f-4353-b9ee-e85751503f91" path="/var/lib/kubelet/pods/573f8be8-460f-4353-b9ee-e85751503f91/volumes" Oct 29 23:03:23.641513 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:23.641488 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ad7ba1-789d-4681-a532-39068330f54f" path="/var/lib/kubelet/pods/b8ad7ba1-789d-4681-a532-39068330f54f/volumes" Oct 29 23:03:24.481241 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:24.481207 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:25.810809 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:25.810445 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:25.976816 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:25.976754 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:26.162273 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:26.162185 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:26.939792 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:26.939756 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:27.109972 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:27.109938 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:27.878423 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:27.875568 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:28.339912 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:28.339872 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:28.679618 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:28.679530 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:29.176417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:29.159142 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:29.613872 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:29.613830 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:30.343301 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:30.343252 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:32.357067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:32.356989 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:32.560405 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:32.560346 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:32.719648 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:32.719619 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:33.811343 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:33.811298 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt"] Oct 29 23:03:33.811824 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:33.811552 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" podUID="4dd1953c-f0a8-4353-bfe3-55e3d25c4212" containerName="maas-api" containerID="cri-o://1f118e4053937b1bc3092ae63324df924c3366355eae5ba5f21447d1a152fec4" gracePeriod=30 Oct 29 23:03:33.918742 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:33.918700 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:34.145474 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.134751 5610 generic.go:356] "Generic (PLEG): container finished" podID="4dd1953c-f0a8-4353-bfe3-55e3d25c4212" containerID="1f118e4053937b1bc3092ae63324df924c3366355eae5ba5f21447d1a152fec4" exitCode=0 Oct 29 23:03:34.145474 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.134868 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" event={"ID":"4dd1953c-f0a8-4353-bfe3-55e3d25c4212","Type":"ContainerDied","Data":"1f118e4053937b1bc3092ae63324df924c3366355eae5ba5f21447d1a152fec4"} Oct 29 23:03:34.240234 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.240203 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 23:03:34.367038 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.366997 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-openshift-service-ca\") pod \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " Oct 29 23:03:34.367324 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.367305 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-metrics-certs\") pod \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " Oct 29 23:03:34.367486 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.367471 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-kube-api-access-8chz2\" (UniqueName: \"kubernetes.io/projected/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-kube-api-access-8chz2\") pod \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " Oct 29 23:03:34.367856 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.367837 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-maas-api-tls\") pod \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\" (UID: \"4dd1953c-f0a8-4353-bfe3-55e3d25c4212\") " Oct 29 23:03:34.370043 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.370019 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-openshift-service-ca" pod "4dd1953c-f0a8-4353-bfe3-55e3d25c4212" (UID: "4dd1953c-f0a8-4353-bfe3-55e3d25c4212"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:34.377200 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.377160 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-metrics-certs" pod "4dd1953c-f0a8-4353-bfe3-55e3d25c4212" (UID: "4dd1953c-f0a8-4353-bfe3-55e3d25c4212"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:34.379224 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.379039 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-maas-api-tls" pod "4dd1953c-f0a8-4353-bfe3-55e3d25c4212" (UID: "4dd1953c-f0a8-4353-bfe3-55e3d25c4212"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:34.379224 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.379191 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-kube-api-access-8chz2" pod "4dd1953c-f0a8-4353-bfe3-55e3d25c4212" (UID: "4dd1953c-f0a8-4353-bfe3-55e3d25c4212"). InnerVolumeSpecName "kube-api-access-8chz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:34.470310 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.470269 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:34.470310 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.470305 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:34.470310 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.470317 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chz2\" (UniqueName: \"kubernetes.io/projected/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-kube-api-access-8chz2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:34.470660 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.470329 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd1953c-f0a8-4353-bfe3-55e3d25c4212-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:34.577181 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.577143 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:34.733226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.733190 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:34.897866 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:34.897828 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:35.063255 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.063170 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:35.144092 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.144066 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" Oct 29 23:03:35.144281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.144088 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt" event={"ID":"4dd1953c-f0a8-4353-bfe3-55e3d25c4212","Type":"ContainerDied","Data":"2783abd20d7e5ba43985d3348eb1f8da0c3b904acd0f734b327f6877b188e02a"} Oct 29 23:03:35.144281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.144157 5610 scope.go:122] "RemoveContainer" containerID="1f118e4053937b1bc3092ae63324df924c3366355eae5ba5f21447d1a152fec4" Oct 29 23:03:35.301732 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.301440 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc"] Oct 29 23:03:35.302653 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302612 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3940cff7-5a19-4fc8-9c07-9f8233186f2e" containerName="maas-api" Oct 29 23:03:35.302653 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302641 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3940cff7-5a19-4fc8-9c07-9f8233186f2e" containerName="maas-api" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302669 5610 cpu_manager.go:414] "containerMap: removing container" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302678 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302712 5610 cpu_manager.go:414] "containerMap: removing container" podUID="b8ad7ba1-789d-4681-a532-39068330f54f" containerName="payload-pre-processing" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302721 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ad7ba1-789d-4681-a532-39068330f54f" containerName="payload-pre-processing" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302740 5610 cpu_manager.go:414] "containerMap: removing container" podUID="7e826947-7906-41b9-9bd5-2147dbb0c076" containerName="payload-processing" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302748 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e826947-7906-41b9-9bd5-2147dbb0c076" containerName="payload-processing" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302763 5610 cpu_manager.go:414] "containerMap: removing container" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302770 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302789 5610 cpu_manager.go:414] "containerMap: removing container" podUID="4dd1953c-f0a8-4353-bfe3-55e3d25c4212" containerName="maas-api" Oct 29 23:03:35.302943 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302797 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd1953c-f0a8-4353-bfe3-55e3d25c4212" containerName="maas-api" Oct 29 23:03:35.303510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.302997 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" Oct 29 23:03:35.303510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.303022 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ad7ba1-789d-4681-a532-39068330f54f" containerName="payload-pre-processing" Oct 29 23:03:35.303510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.303037 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e826947-7906-41b9-9bd5-2147dbb0c076" containerName="payload-processing" Oct 29 23:03:35.303510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.303053 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd1953c-f0a8-4353-bfe3-55e3d25c4212" containerName="maas-api" Oct 29 23:03:35.303510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.303070 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3940cff7-5a19-4fc8-9c07-9f8233186f2e" containerName="maas-api" Oct 29 23:03:35.360738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.330569 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:35.360738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.338759 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efc363-2b9f-48c8-95dc-962e7aade523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc\" (UID: \"69efc363-2b9f-48c8-95dc-962e7aade523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:35.360738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.339007 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z677g\" (UniqueName: \"kubernetes.io/projected/69efc363-2b9f-48c8-95dc-962e7aade523-kube-api-access-z677g\") pod \"payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc\" (UID: \"69efc363-2b9f-48c8-95dc-962e7aade523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:35.360738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.344983 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c74e55-dockercfg-q82fw\"" Oct 29 23:03:35.360738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.345601 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c74e55\"" Oct 29 23:03:35.371728 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.371697 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt"] Oct 29 23:03:35.377011 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.376984 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8d3632-54b4fbf5f8-5szqt"] Oct 29 23:03:35.382643 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.382622 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc"] Oct 29 23:03:35.440102 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.440070 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z677g\" (UniqueName: \"kubernetes.io/projected/69efc363-2b9f-48c8-95dc-962e7aade523-kube-api-access-z677g\") pod \"payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc\" (UID: \"69efc363-2b9f-48c8-95dc-962e7aade523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:35.440283 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.440202 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efc363-2b9f-48c8-95dc-962e7aade523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc\" (UID: \"69efc363-2b9f-48c8-95dc-962e7aade523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:35.440934 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.440914 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efc363-2b9f-48c8-95dc-962e7aade523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc\" (UID: \"69efc363-2b9f-48c8-95dc-962e7aade523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:35.459239 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.459202 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z677g\" (UniqueName: \"kubernetes.io/projected/69efc363-2b9f-48c8-95dc-962e7aade523-kube-api-access-z677g\") pod \"payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc\" (UID: \"69efc363-2b9f-48c8-95dc-962e7aade523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:35.488248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.488214 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:35.637208 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.637115 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd1953c-f0a8-4353-bfe3-55e3d25c4212" path="/var/lib/kubelet/pods/4dd1953c-f0a8-4353-bfe3-55e3d25c4212/volumes" Oct 29 23:03:35.661548 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.661517 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:35.969736 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:35.968537 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc"] Oct 29 23:03:35.969736 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:03:35.968893 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69efc363_2b9f_48c8_95dc_962e7aade523.slice/crio-55d0f8ca437712c9b0290c330e8c0cbc6f6bd7fd2d1a104584f450e9acea17f0 WatchSource:0}: Error finding container 55d0f8ca437712c9b0290c330e8c0cbc6f6bd7fd2d1a104584f450e9acea17f0: Status 404 returned error can't find the container with id 55d0f8ca437712c9b0290c330e8c0cbc6f6bd7fd2d1a104584f450e9acea17f0 Oct 29 23:03:36.154712 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:36.153928 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" event={"ID":"69efc363-2b9f-48c8-95dc-962e7aade523","Type":"ContainerStarted","Data":"ae1c30834d1b5a566e0284aae08216d105e0f986c348e032cb39e348033724ac"} Oct 29 23:03:36.154712 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:36.153999 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:36.154712 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:36.154024 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" event={"ID":"69efc363-2b9f-48c8-95dc-962e7aade523","Type":"ContainerStarted","Data":"55d0f8ca437712c9b0290c330e8c0cbc6f6bd7fd2d1a104584f450e9acea17f0"} Oct 29 23:03:36.202074 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:36.202022 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" podStartSLOduration=1.202008648 podStartE2EDuration="1.202008648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:03:36.199867672 +0000 UTC m=+2878.681185165" watchObservedRunningTime="2026-10-29 23:03:36.202008648 +0000 UTC m=+2878.683326142" Oct 29 23:03:36.837790 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:36.837749 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:38.236683 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:38.236635 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:39.728422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:39.710655 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:40.947436 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:40.947365 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.570216 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:41.570180 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.722355 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:41.722314 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:41.931188 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:41.931076 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:42.106994 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:42.106944 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:43.713069 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:43.713031 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:03:43.952230 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:43.943956 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:45.306424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:45.300210 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:46.902004 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:46.901968 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:47.167271 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:47.167181 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:03:48.362386 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:48.362303 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:49.747450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:49.747408 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:51.289419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:51.289366 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:51.431524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:51.431493 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:51.570961 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:51.570774 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:52.547348 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.547311 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:52.577892 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.577852 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc"] Oct 29 23:03:52.578175 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.578133 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" podUID="0c1f6332-9b86-44ba-84f7-21c12c669084" containerName="payload-processing" containerID="cri-o://43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406" gracePeriod=30 Oct 29 23:03:52.601512 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.601473 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7"] Oct 29 23:03:52.601691 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.601662 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" podUID="5da553dc-3f75-4297-97c4-0ca75b6379c9" containerName="payload-pre-processing" containerID="cri-o://759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e" gracePeriod=30 Oct 29 23:03:52.691601 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.691550 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:52.819966 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.812097 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s"] Oct 29 23:03:52.819966 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.812362 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" podUID="2bad0630-e43a-48c3-bd90-aecc085ba292" containerName="maas-api" containerID="cri-o://db2268d564111670a532d3a419fe482a2af97ade2ea3aaa7071ad9220512334c" gracePeriod=30 Oct 29 23:03:52.846668 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.846531 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:52.924311 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.924285 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:03:52.990418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.973182 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:03:52.990418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.989272 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz"] Oct 29 23:03:52.990418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:52.989513 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" podUID="cd12124e-237a-49c3-add4-2cfd0f83b589" containerName="payload-pre-processing" containerID="cri-o://0b6b6c1ed80080205f6efdfb56a529faa5a7214e9c1a6713326f969e1f1ddc77" gracePeriod=30 Oct 29 23:03:53.009473 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.009438 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1f6332-9b86-44ba-84f7-21c12c669084-kube-api-access-rdcnz\" (UniqueName: \"kubernetes.io/projected/0c1f6332-9b86-44ba-84f7-21c12c669084-kube-api-access-rdcnz\") pod \"0c1f6332-9b86-44ba-84f7-21c12c669084\" (UID: \"0c1f6332-9b86-44ba-84f7-21c12c669084\") " Oct 29 23:03:53.009651 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.009566 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1f6332-9b86-44ba-84f7-21c12c669084-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1f6332-9b86-44ba-84f7-21c12c669084-plugins-config-volume\") pod \"0c1f6332-9b86-44ba-84f7-21c12c669084\" (UID: \"0c1f6332-9b86-44ba-84f7-21c12c669084\") " Oct 29 23:03:53.022910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.010487 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1f6332-9b86-44ba-84f7-21c12c669084-plugins-config-volume" pod "0c1f6332-9b86-44ba-84f7-21c12c669084" (UID: "0c1f6332-9b86-44ba-84f7-21c12c669084"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:53.022910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.011693 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1f6332-9b86-44ba-84f7-21c12c669084-kube-api-access-rdcnz" pod "0c1f6332-9b86-44ba-84f7-21c12c669084" (UID: "0c1f6332-9b86-44ba-84f7-21c12c669084"). InnerVolumeSpecName "kube-api-access-rdcnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:53.026780 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.026637 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:53.111876 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.111840 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da553dc-3f75-4297-97c4-0ca75b6379c9-kube-api-access-4mkkw\" (UniqueName: \"kubernetes.io/projected/5da553dc-3f75-4297-97c4-0ca75b6379c9-kube-api-access-4mkkw\") pod \"5da553dc-3f75-4297-97c4-0ca75b6379c9\" (UID: \"5da553dc-3f75-4297-97c4-0ca75b6379c9\") " Oct 29 23:03:53.112099 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.111981 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da553dc-3f75-4297-97c4-0ca75b6379c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da553dc-3f75-4297-97c4-0ca75b6379c9-plugins-config-volume\") pod \"5da553dc-3f75-4297-97c4-0ca75b6379c9\" (UID: \"5da553dc-3f75-4297-97c4-0ca75b6379c9\") " Oct 29 23:03:53.112435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.112406 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da553dc-3f75-4297-97c4-0ca75b6379c9-plugins-config-volume" pod "5da553dc-3f75-4297-97c4-0ca75b6379c9" (UID: "5da553dc-3f75-4297-97c4-0ca75b6379c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:53.112622 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.112604 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da553dc-3f75-4297-97c4-0ca75b6379c9-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.112704 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.112629 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcnz\" (UniqueName: \"kubernetes.io/projected/0c1f6332-9b86-44ba-84f7-21c12c669084-kube-api-access-rdcnz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.112704 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.112645 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1f6332-9b86-44ba-84f7-21c12c669084-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.114690 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.114666 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da553dc-3f75-4297-97c4-0ca75b6379c9-kube-api-access-4mkkw" pod "5da553dc-3f75-4297-97c4-0ca75b6379c9" (UID: "5da553dc-3f75-4297-97c4-0ca75b6379c9"). InnerVolumeSpecName "kube-api-access-4mkkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:53.182956 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.182837 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:53.213805 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.213737 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mkkw\" (UniqueName: \"kubernetes.io/projected/5da553dc-3f75-4297-97c4-0ca75b6379c9-kube-api-access-4mkkw\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.483048 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.482526 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:53.488853 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.484015 5610 generic.go:356] "Generic (PLEG): container finished" podID="5da553dc-3f75-4297-97c4-0ca75b6379c9" containerID="759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e" exitCode=0 Oct 29 23:03:53.488853 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.484115 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" event={"ID":"5da553dc-3f75-4297-97c4-0ca75b6379c9","Type":"ContainerDied","Data":"759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e"} Oct 29 23:03:53.488853 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.484163 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" event={"ID":"5da553dc-3f75-4297-97c4-0ca75b6379c9","Type":"ContainerDied","Data":"0c50dacccb0cc653a61ce56525345a4c3fd50e04a5d3f48a8348942eb9b63f08"} Oct 29 23:03:53.488853 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.484191 5610 scope.go:122] "RemoveContainer" containerID="759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e" Oct 29 23:03:53.488853 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.484349 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7" Oct 29 23:03:53.498514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.497654 5610 generic.go:356] "Generic (PLEG): container finished" podID="2bad0630-e43a-48c3-bd90-aecc085ba292" containerID="db2268d564111670a532d3a419fe482a2af97ade2ea3aaa7071ad9220512334c" exitCode=0 Oct 29 23:03:53.498514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.497816 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" event={"ID":"2bad0630-e43a-48c3-bd90-aecc085ba292","Type":"ContainerDied","Data":"db2268d564111670a532d3a419fe482a2af97ade2ea3aaa7071ad9220512334c"} Oct 29 23:03:53.500501 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.500448 5610 generic.go:356] "Generic (PLEG): container finished" podID="cd12124e-237a-49c3-add4-2cfd0f83b589" containerID="0b6b6c1ed80080205f6efdfb56a529faa5a7214e9c1a6713326f969e1f1ddc77" exitCode=0 Oct 29 23:03:53.500682 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.500658 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" event={"ID":"cd12124e-237a-49c3-add4-2cfd0f83b589","Type":"ContainerDied","Data":"0b6b6c1ed80080205f6efdfb56a529faa5a7214e9c1a6713326f969e1f1ddc77"} Oct 29 23:03:53.502609 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.502584 5610 generic.go:356] "Generic (PLEG): container finished" podID="0c1f6332-9b86-44ba-84f7-21c12c669084" containerID="43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406" exitCode=0 Oct 29 23:03:53.503372 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.502834 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 23:03:53.503794 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.503275 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" Oct 29 23:03:53.504724 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.503291 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" event={"ID":"0c1f6332-9b86-44ba-84f7-21c12c669084","Type":"ContainerDied","Data":"43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406"} Oct 29 23:03:53.504724 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.504620 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc" event={"ID":"0c1f6332-9b86-44ba-84f7-21c12c669084","Type":"ContainerDied","Data":"df8013e306bcf6859b8db39fc5ed5885dc2c6c354af8e59456d5383449324603"} Oct 29 23:03:53.510436 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.510151 5610 scope.go:122] "RemoveContainer" containerID="759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e" Oct 29 23:03:53.510556 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:53.510521 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e\": container with ID starting with 759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e not found: ID does not exist" containerID="759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e" Oct 29 23:03:53.510630 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.510554 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e"} err="failed to get container status \"759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e\": rpc error: code = NotFound desc = could not find container \"759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e\": container with ID starting with 759b1f7e9596b5d82c5c0bb751cbec5b27e3513a0315b736b221cc08d0969e2e not found: ID does not exist" Oct 29 23:03:53.510630 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.510579 5610 scope.go:122] "RemoveContainer" containerID="43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406" Oct 29 23:03:53.526283 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.526032 5610 scope.go:122] "RemoveContainer" containerID="43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406" Oct 29 23:03:53.527127 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:03:53.527099 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406\": container with ID starting with 43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406 not found: ID does not exist" containerID="43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406" Oct 29 23:03:53.527215 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.527136 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406"} err="failed to get container status \"43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406\": rpc error: code = NotFound desc = could not find container \"43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406\": container with ID starting with 43432fa89511cfb24f4a55390d8b5059d8cb773cb0eabfd617bab4a8be205406 not found: ID does not exist" Oct 29 23:03:53.620193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.620109 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7"] Oct 29 23:03:53.626622 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.626594 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d286f-5bb646bf59-8tzh7"] Oct 29 23:03:53.667334 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.666215 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da553dc-3f75-4297-97c4-0ca75b6379c9" path="/var/lib/kubelet/pods/5da553dc-3f75-4297-97c4-0ca75b6379c9/volumes" Oct 29 23:03:53.669999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.667609 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd12124e-237a-49c3-add4-2cfd0f83b589-kube-api-access-mg9bt\" (UniqueName: \"kubernetes.io/projected/cd12124e-237a-49c3-add4-2cfd0f83b589-kube-api-access-mg9bt\") pod \"cd12124e-237a-49c3-add4-2cfd0f83b589\" (UID: \"cd12124e-237a-49c3-add4-2cfd0f83b589\") " Oct 29 23:03:53.669999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.667742 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd12124e-237a-49c3-add4-2cfd0f83b589-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd12124e-237a-49c3-add4-2cfd0f83b589-plugins-config-volume\") pod \"cd12124e-237a-49c3-add4-2cfd0f83b589\" (UID: \"cd12124e-237a-49c3-add4-2cfd0f83b589\") " Oct 29 23:03:53.669999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.669098 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd12124e-237a-49c3-add4-2cfd0f83b589-plugins-config-volume" pod "cd12124e-237a-49c3-add4-2cfd0f83b589" (UID: "cd12124e-237a-49c3-add4-2cfd0f83b589"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:53.672008 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.671942 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd12124e-237a-49c3-add4-2cfd0f83b589-kube-api-access-mg9bt" pod "cd12124e-237a-49c3-add4-2cfd0f83b589" (UID: "cd12124e-237a-49c3-add4-2cfd0f83b589"). InnerVolumeSpecName "kube-api-access-mg9bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:53.678852 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.678822 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:53.679560 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.679533 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc"] Oct 29 23:03:53.680183 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.680165 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 23:03:53.686459 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.686437 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d286f-79f5f996b4-lk6fc"] Oct 29 23:03:53.773406 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.770207 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bad0630-e43a-48c3-bd90-aecc085ba292-kube-api-access-bqjxl\" (UniqueName: \"kubernetes.io/projected/2bad0630-e43a-48c3-bd90-aecc085ba292-kube-api-access-bqjxl\") pod \"2bad0630-e43a-48c3-bd90-aecc085ba292\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " Oct 29 23:03:53.773406 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.770861 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bad0630-e43a-48c3-bd90-aecc085ba292-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad0630-e43a-48c3-bd90-aecc085ba292-openshift-service-ca\") pod \"2bad0630-e43a-48c3-bd90-aecc085ba292\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " Oct 29 23:03:53.773406 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.770980 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-metrics-certs\") pod \"2bad0630-e43a-48c3-bd90-aecc085ba292\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " Oct 29 23:03:53.773406 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.771203 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-maas-api-tls\") pod \"2bad0630-e43a-48c3-bd90-aecc085ba292\" (UID: \"2bad0630-e43a-48c3-bd90-aecc085ba292\") " Oct 29 23:03:53.773406 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.771812 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bad0630-e43a-48c3-bd90-aecc085ba292-openshift-service-ca" pod "2bad0630-e43a-48c3-bd90-aecc085ba292" (UID: "2bad0630-e43a-48c3-bd90-aecc085ba292"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:03:53.773406 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.773061 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bad0630-e43a-48c3-bd90-aecc085ba292-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.773406 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.773099 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg9bt\" (UniqueName: \"kubernetes.io/projected/cd12124e-237a-49c3-add4-2cfd0f83b589-kube-api-access-mg9bt\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.773406 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.773117 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd12124e-237a-49c3-add4-2cfd0f83b589-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.790533 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.790480 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bad0630-e43a-48c3-bd90-aecc085ba292-kube-api-access-bqjxl" pod "2bad0630-e43a-48c3-bd90-aecc085ba292" (UID: "2bad0630-e43a-48c3-bd90-aecc085ba292"). InnerVolumeSpecName "kube-api-access-bqjxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:03:53.790714 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.790528 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-maas-api-tls" pod "2bad0630-e43a-48c3-bd90-aecc085ba292" (UID: "2bad0630-e43a-48c3-bd90-aecc085ba292"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:53.792650 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.792620 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-metrics-certs" pod "2bad0630-e43a-48c3-bd90-aecc085ba292" (UID: "2bad0630-e43a-48c3-bd90-aecc085ba292"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:03:53.880950 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.879789 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqjxl\" (UniqueName: \"kubernetes.io/projected/2bad0630-e43a-48c3-bd90-aecc085ba292-kube-api-access-bqjxl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.880950 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.879830 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:53.880950 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:53.879844 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2bad0630-e43a-48c3-bd90-aecc085ba292-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:03:54.108089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.108005 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:54.332672 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.332639 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:54.515913 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.515880 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" Oct 29 23:03:54.516114 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.515906 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s" event={"ID":"2bad0630-e43a-48c3-bd90-aecc085ba292","Type":"ContainerDied","Data":"5e4b37a0c2ebaf4c5237154566763c649b4b4c369e6a0df0da6883b0e87025f2"} Oct 29 23:03:54.516114 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.515967 5610 scope.go:122] "RemoveContainer" containerID="db2268d564111670a532d3a419fe482a2af97ade2ea3aaa7071ad9220512334c" Oct 29 23:03:54.517698 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.517674 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" Oct 29 23:03:54.517852 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.517682 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz" event={"ID":"cd12124e-237a-49c3-add4-2cfd0f83b589","Type":"ContainerDied","Data":"97869ca412919502aad8eb91f0a33ce70af4b4602e63d4ac9bb7a0d9ee043e84"} Oct 29 23:03:54.531784 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.531760 5610 scope.go:122] "RemoveContainer" containerID="0b6b6c1ed80080205f6efdfb56a529faa5a7214e9c1a6713326f969e1f1ddc77" Oct 29 23:03:54.579377 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.579348 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s"] Oct 29 23:03:54.590776 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.590749 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1dabb8-7654fbc69c-9p46s"] Oct 29 23:03:54.616696 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.616668 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz"] Oct 29 23:03:54.619908 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:54.619883 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dabb8-7f8cb6f4c9-9xxxz"] Oct 29 23:03:55.637477 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:55.637446 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1f6332-9b86-44ba-84f7-21c12c669084" path="/var/lib/kubelet/pods/0c1f6332-9b86-44ba-84f7-21c12c669084/volumes" Oct 29 23:03:55.637920 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:55.637906 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bad0630-e43a-48c3-bd90-aecc085ba292" path="/var/lib/kubelet/pods/2bad0630-e43a-48c3-bd90-aecc085ba292/volumes" Oct 29 23:03:55.638280 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:55.638268 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd12124e-237a-49c3-add4-2cfd0f83b589" path="/var/lib/kubelet/pods/cd12124e-237a-49c3-add4-2cfd0f83b589/volumes" Oct 29 23:03:55.705415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:55.704884 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:57.143023 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:57.142989 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:58.833415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:58.833044 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:59.023006 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:59.022957 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:03:59.280081 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:03:59.280022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:00.049763 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:00.049730 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:00.273122 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:00.273086 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l"] Oct 29 23:04:00.273417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:00.273373 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" containerID="cri-o://31a0f05c93d788d20ca3c1cc6e8c8ea693b48bb673437ace892029712a8e9725" gracePeriod=30 Oct 29 23:04:00.278090 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:00.277281 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:00.455475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:00.435057 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:00.852220 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:00.852121 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:02.472827 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.472788 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8"] Oct 29 23:04:02.473936 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.473916 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0c1f6332-9b86-44ba-84f7-21c12c669084" containerName="payload-processing" Oct 29 23:04:02.474011 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.473939 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1f6332-9b86-44ba-84f7-21c12c669084" containerName="payload-processing" Oct 29 23:04:02.474011 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.473962 5610 cpu_manager.go:414] "containerMap: removing container" podUID="cd12124e-237a-49c3-add4-2cfd0f83b589" containerName="payload-pre-processing" Oct 29 23:04:02.474011 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.473973 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd12124e-237a-49c3-add4-2cfd0f83b589" containerName="payload-pre-processing" Oct 29 23:04:02.474011 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.473997 5610 cpu_manager.go:414] "containerMap: removing container" podUID="5da553dc-3f75-4297-97c4-0ca75b6379c9" containerName="payload-pre-processing" Oct 29 23:04:02.474011 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.474005 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da553dc-3f75-4297-97c4-0ca75b6379c9" containerName="payload-pre-processing" Oct 29 23:04:02.474176 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.474044 5610 cpu_manager.go:414] "containerMap: removing container" podUID="2bad0630-e43a-48c3-bd90-aecc085ba292" containerName="maas-api" Oct 29 23:04:02.474176 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.474051 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bad0630-e43a-48c3-bd90-aecc085ba292" containerName="maas-api" Oct 29 23:04:02.474248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.474236 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bad0630-e43a-48c3-bd90-aecc085ba292" containerName="maas-api" Oct 29 23:04:02.474286 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.474255 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="573f8be8-460f-4353-b9ee-e85751503f91" containerName="istio-proxy" Oct 29 23:04:02.474286 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.474269 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1f6332-9b86-44ba-84f7-21c12c669084" containerName="payload-processing" Oct 29 23:04:02.474286 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.474282 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd12124e-237a-49c3-add4-2cfd0f83b589" containerName="payload-pre-processing" Oct 29 23:04:02.474380 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.474293 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da553dc-3f75-4297-97c4-0ca75b6379c9" containerName="payload-pre-processing" Oct 29 23:04:02.480289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.480266 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.501084 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.500380 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8"] Oct 29 23:04:02.505105 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.505082 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c74e55-serving-cert\"" Oct 29 23:04:02.633769 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.633741 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.633952 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.633791 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.634004 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.633965 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.634102 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.634080 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5nz\" (UniqueName: \"kubernetes.io/projected/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-kube-api-access-ql5nz\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.695832 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.695784 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:02.734982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.734883 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.735174 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.735031 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5nz\" (UniqueName: \"kubernetes.io/projected/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-kube-api-access-ql5nz\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.735432 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.735386 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.735698 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.735668 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.736288 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.736265 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.737967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.737944 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.738061 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.738023 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.754404 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.754367 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5nz\" (UniqueName: \"kubernetes.io/projected/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-kube-api-access-ql5nz\") pod \"maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.806338 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.806305 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:02.846695 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:02.846656 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:03.121137 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:03.121101 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8"] Oct 29 23:04:03.123103 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:04:03.123064 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eee6ddd_7369_4268_b2fb_74c11ed01e3d.slice/crio-3c005f2a4d9502f017154a40e51b3102f97b51030d739f94b18e35832bae7f06 WatchSource:0}: Error finding container 3c005f2a4d9502f017154a40e51b3102f97b51030d739f94b18e35832bae7f06: Status 404 returned error can't find the container with id 3c005f2a4d9502f017154a40e51b3102f97b51030d739f94b18e35832bae7f06 Oct 29 23:04:03.593485 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:03.593442 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" event={"ID":"2eee6ddd-7369-4268-b2fb-74c11ed01e3d","Type":"ContainerStarted","Data":"0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740"} Oct 29 23:04:03.594010 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:03.593499 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" event={"ID":"2eee6ddd-7369-4268-b2fb-74c11ed01e3d","Type":"ContainerStarted","Data":"3c005f2a4d9502f017154a40e51b3102f97b51030d739f94b18e35832bae7f06"} Oct 29 23:04:03.594010 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:03.593528 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:03.637596 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:03.637546 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" podStartSLOduration=1.309166658 podStartE2EDuration="1.63752856s" totalImagesPullingTime="328.361902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:03.635913853 +0000 UTC m=+2906.117231347" watchObservedRunningTime="2026-10-29 23:04:03.63752856 +0000 UTC m=+2906.118846054" Oct 29 23:04:04.350405 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:04.350350 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:05.608837 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:05.608756 5610 generic.go:356] "Generic (PLEG): container finished" podID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerID="31a0f05c93d788d20ca3c1cc6e8c8ea693b48bb673437ace892029712a8e9725" exitCode=0 Oct 29 23:04:05.609243 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:05.608832 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" event={"ID":"0e83482a-0401-47df-ab5a-1b419b3c371d","Type":"ContainerDied","Data":"31a0f05c93d788d20ca3c1cc6e8c8ea693b48bb673437ace892029712a8e9725"} Oct 29 23:04:05.609243 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:05.608892 5610 scope.go:122] "RemoveContainer" containerID="44d98ba7967fcd6dfc9c63a27f14cee4be9ee087e3153fdc061607c23f1f071e" Oct 29 23:04:05.960525 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:05.960352 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:05.985199 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:05.985166 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:04:06.080222 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080183 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-envoy\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080411 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080292 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-socket\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080411 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080340 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-token\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-token\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080561 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080418 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-certs\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080561 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080522 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-podinfo\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080687 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080668 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-data\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080754 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080727 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-certs" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:06.080820 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080751 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-kube-api-access-x6l5q\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-kube-api-access-x6l5q\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080820 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080791 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e83482a-0401-47df-ab5a-1b419b3c371d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e83482a-0401-47df-ab5a-1b419b3c371d-istiod-ca-cert\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080934 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080853 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-credential-socket\") pod \"0e83482a-0401-47df-ab5a-1b419b3c371d\" (UID: \"0e83482a-0401-47df-ab5a-1b419b3c371d\") " Oct 29 23:04:06.080992 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.080935 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-data" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:06.081184 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.081162 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-credential-socket" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:06.081274 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.081193 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e83482a-0401-47df-ab5a-1b419b3c371d-istiod-ca-cert" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:06.081374 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.081331 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-socket" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:06.081626 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.081605 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.081678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.081635 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.081678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.081651 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.081678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.081666 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e83482a-0401-47df-ab5a-1b419b3c371d-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.081784 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.081680 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.083620 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.083554 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-podinfo" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:06.083620 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.083589 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-envoy" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:06.083793 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.083726 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-token" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:06.085138 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.085119 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-kube-api-access-x6l5q" pod "0e83482a-0401-47df-ab5a-1b419b3c371d" (UID: "0e83482a-0401-47df-ab5a-1b419b3c371d"). InnerVolumeSpecName "kube-api-access-x6l5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:06.183448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.183339 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.183448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.183376 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6l5q\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-kube-api-access-x6l5q\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.183448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.183414 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.183448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.183428 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e83482a-0401-47df-ab5a-1b419b3c371d-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:06.620890 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.620858 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" Oct 29 23:04:06.620890 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.620879 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l" event={"ID":"0e83482a-0401-47df-ab5a-1b419b3c371d","Type":"ContainerDied","Data":"40f0d78f0c9c6c41da4f040f1aa65cef147e30e25509cdf596e3a8fa1cc9736b"} Oct 29 23:04:06.621373 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.620976 5610 scope.go:122] "RemoveContainer" containerID="31a0f05c93d788d20ca3c1cc6e8c8ea693b48bb673437ace892029712a8e9725" Oct 29 23:04:06.684632 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.684598 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l"] Oct 29 23:04:06.695286 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:06.695254 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d286f-openshift-default-6c84c45c89-lz28l"] Oct 29 23:04:07.326415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:07.318604 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:07.636828 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:07.636758 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" path="/var/lib/kubelet/pods/0e83482a-0401-47df-ab5a-1b419b3c371d/volumes" Oct 29 23:04:08.908137 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:08.908106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:09.606514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:09.606486 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:04:10.447344 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:10.445828 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:11.230662 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:11.230633 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv"] Oct 29 23:04:11.266457 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:11.266419 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2"] Oct 29 23:04:11.424288 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:11.424234 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:11.563341 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:11.563218 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:12.414864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:12.414827 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:13.717738 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:04:13.717696 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:04:14.580467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:14.580424 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:15.928181 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:15.928147 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:16.231029 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:16.230989 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" podUID="eb67a1af-c1e4-4990-8c95-5a9c11f9f625" containerName="payload-processing" containerID="cri-o://a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645" gracePeriod=25 Oct 29 23:04:16.266801 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:16.266746 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" podUID="9ef0069e-4eb3-4789-a080-b9d4920a5140" containerName="payload-pre-processing" containerID="cri-o://666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f" gracePeriod=25 Oct 29 23:04:17.423462 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:17.420761 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:17.821226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:17.821187 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" podUID="eb67a1af-c1e4-4990-8c95-5a9c11f9f625" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:04:17.821442 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:17.821278 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" podUID="9ef0069e-4eb3-4789-a080-b9d4920a5140" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:04:19.017520 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:19.016283 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:20.062315 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.062255 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd"] Oct 29 23:04:20.062781 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.062508 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" podUID="4474e487-1772-417f-82d8-c52932f1b21a" containerName="maas-api" containerID="cri-o://bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0" gracePeriod=30 Oct 29 23:04:20.172447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.172408 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:20.529435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.528491 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 23:04:20.704339 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.704300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:20.721035 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.720950 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4474e487-1772-417f-82d8-c52932f1b21a-kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/4474e487-1772-417f-82d8-c52932f1b21a-kube-api-access-h8lqg\") pod \"4474e487-1772-417f-82d8-c52932f1b21a\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " Oct 29 23:04:20.721035 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.720998 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4474e487-1772-417f-82d8-c52932f1b21a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474e487-1772-417f-82d8-c52932f1b21a-openshift-service-ca\") pod \"4474e487-1772-417f-82d8-c52932f1b21a\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " Oct 29 23:04:20.721286 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.721037 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-metrics-certs\") pod \"4474e487-1772-417f-82d8-c52932f1b21a\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " Oct 29 23:04:20.721502 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.721472 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4474e487-1772-417f-82d8-c52932f1b21a-openshift-service-ca" pod "4474e487-1772-417f-82d8-c52932f1b21a" (UID: "4474e487-1772-417f-82d8-c52932f1b21a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:20.721618 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.721568 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-maas-api-tls\") pod \"4474e487-1772-417f-82d8-c52932f1b21a\" (UID: \"4474e487-1772-417f-82d8-c52932f1b21a\") " Oct 29 23:04:20.722427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.722365 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4474e487-1772-417f-82d8-c52932f1b21a-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:20.723515 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.723493 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-metrics-certs" pod "4474e487-1772-417f-82d8-c52932f1b21a" (UID: "4474e487-1772-417f-82d8-c52932f1b21a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:20.724444 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.724424 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-maas-api-tls" pod "4474e487-1772-417f-82d8-c52932f1b21a" (UID: "4474e487-1772-417f-82d8-c52932f1b21a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:04:20.724525 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.724505 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4474e487-1772-417f-82d8-c52932f1b21a-kube-api-access-h8lqg" pod "4474e487-1772-417f-82d8-c52932f1b21a" (UID: "4474e487-1772-417f-82d8-c52932f1b21a"). InnerVolumeSpecName "kube-api-access-h8lqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:20.735335 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.735309 5610 generic.go:356] "Generic (PLEG): container finished" podID="4474e487-1772-417f-82d8-c52932f1b21a" containerID="bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0" exitCode=0 Oct 29 23:04:20.735483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.735343 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" event={"ID":"4474e487-1772-417f-82d8-c52932f1b21a","Type":"ContainerDied","Data":"bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0"} Oct 29 23:04:20.735483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.735373 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" Oct 29 23:04:20.735483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.735419 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" event={"ID":"4474e487-1772-417f-82d8-c52932f1b21a","Type":"ContainerDied","Data":"af5fb5f76fe7eec1ad9e91d9d33498d01ff4f7fc2f1d546d7dcf321806bcafc9"} Oct 29 23:04:20.735483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.735441 5610 scope.go:122] "RemoveContainer" containerID="bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0" Oct 29 23:04:20.752644 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.752627 5610 scope.go:122] "RemoveContainer" containerID="bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0" Oct 29 23:04:20.752937 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:04:20.752919 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0\": container with ID starting with bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0 not found: ID does not exist" containerID="bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0" Oct 29 23:04:20.752980 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.752950 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0"} err="failed to get container status \"bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0\": rpc error: code = NotFound desc = could not find container \"bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0\": container with ID starting with bae329d1dddb81c829a9f95ccef44967fdb1293f7af8a7ffa6d5700d109069d0 not found: ID does not exist" Oct 29 23:04:20.787206 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.787158 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd"] Oct 29 23:04:20.791725 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.791698 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd"] Oct 29 23:04:20.828012 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.827968 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:20.828012 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.828014 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8lqg\" (UniqueName: \"kubernetes.io/projected/4474e487-1772-417f-82d8-c52932f1b21a-kube-api-access-h8lqg\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:20.828349 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.828030 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4474e487-1772-417f-82d8-c52932f1b21a-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:20.851168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:20.851137 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:21.318559 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:21.318519 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:21.636581 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:21.636507 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4474e487-1772-417f-82d8-c52932f1b21a" path="/var/lib/kubelet/pods/4474e487-1772-417f-82d8-c52932f1b21a/volumes" Oct 29 23:04:22.356026 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:22.355979 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:23.379159 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:23.379115 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-437981-857c8d6bfd-bsxfd" podUID="4474e487-1772-417f-82d8-c52932f1b21a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.174:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 29 23:04:24.091408 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:24.091355 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:25.500653 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:25.500612 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:27.035872 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.035830 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:27.184807 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.184773 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:27.326735 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.326622 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:27.482909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.482868 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:27.661818 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.661733 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw"] Oct 29 23:04:27.662009 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.661986 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" containerID="cri-o://34f377e425dfb7b9c92ddbfeff28c0aebfd7180b4e3bb2cd43d2150e5d39132b" gracePeriod=30 Oct 29 23:04:27.739491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.739455 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:27.830573 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.830036 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" podUID="9ef0069e-4eb3-4789-a080-b9d4920a5140" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:04:27.831434 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:27.831365 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" podUID="eb67a1af-c1e4-4990-8c95-5a9c11f9f625" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:04:29.139880 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:29.139844 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:30.730277 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:30.713094 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:32.341097 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:32.340216 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:32.831662 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:32.831631 5610 generic.go:356] "Generic (PLEG): container finished" podID="8ea5c830-31e9-498e-a774-9c3400e83178" containerID="34f377e425dfb7b9c92ddbfeff28c0aebfd7180b4e3bb2cd43d2150e5d39132b" exitCode=0 Oct 29 23:04:32.831857 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:32.831705 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" event={"ID":"8ea5c830-31e9-498e-a774-9c3400e83178","Type":"ContainerDied","Data":"34f377e425dfb7b9c92ddbfeff28c0aebfd7180b4e3bb2cd43d2150e5d39132b"} Oct 29 23:04:32.831857 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:32.831767 5610 scope.go:122] "RemoveContainer" containerID="6162de3e1c77f15f766b1d4af933e529ff9e8eca9a21d2b6d8cd6c4dd6b7afca" Oct 29 23:04:32.948563 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:32.948535 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 23:04:33.107738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.107698 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea5c830-31e9-498e-a774-9c3400e83178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea5c830-31e9-498e-a774-9c3400e83178-istiod-ca-cert\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.108089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.108063 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-certs\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.108318 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.108283 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-envoy\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.108455 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.108115 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea5c830-31e9-498e-a774-9c3400e83178-istiod-ca-cert" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:33.108551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.108388 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-certs" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.108965 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.108666 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-credential-socket\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.109190 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.108938 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-credential-socket" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.109347 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.109332 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-kube-api-access-6krvt\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-kube-api-access-6krvt\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.109773 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.109758 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-data\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.110176 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.110072 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-data" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.110504 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.110469 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-socket" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.110721 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.110591 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-socket\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.111734 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.111670 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-istio-token\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.111734 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.111569 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-envoy" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:33.111734 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.111711 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ea5c830-31e9-498e-a774-9c3400e83178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea5c830-31e9-498e-a774-9c3400e83178-istio-podinfo\") pod \"8ea5c830-31e9-498e-a774-9c3400e83178\" (UID: \"8ea5c830-31e9-498e-a774-9c3400e83178\") " Oct 29 23:04:33.112451 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.112431 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.112566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.112457 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.112566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.112473 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea5c830-31e9-498e-a774-9c3400e83178-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.112566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.112487 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.112566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.112499 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.112566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.112511 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea5c830-31e9-498e-a774-9c3400e83178-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.115359 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.115334 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-istio-token" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:33.115477 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.115362 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ea5c830-31e9-498e-a774-9c3400e83178-istio-podinfo" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:33.115477 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.115427 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-kube-api-access-6krvt" pod "8ea5c830-31e9-498e-a774-9c3400e83178" (UID: "8ea5c830-31e9-498e-a774-9c3400e83178"). InnerVolumeSpecName "kube-api-access-6krvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:33.213514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.213429 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krvt\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-kube-api-access-6krvt\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.213514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.213459 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea5c830-31e9-498e-a774-9c3400e83178-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.213514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.213476 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea5c830-31e9-498e-a774-9c3400e83178-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:33.742278 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.742246 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc"] Oct 29 23:04:33.851885 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.851849 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" Oct 29 23:04:33.852078 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.851968 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw" event={"ID":"8ea5c830-31e9-498e-a774-9c3400e83178","Type":"ContainerDied","Data":"4580e8d5996fdae461692eed8552aa52e65c528b32ac4b4390ae7532e65479f0"} Oct 29 23:04:33.852078 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.852028 5610 scope.go:122] "RemoveContainer" containerID="34f377e425dfb7b9c92ddbfeff28c0aebfd7180b4e3bb2cd43d2150e5d39132b" Oct 29 23:04:33.911206 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.909589 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw"] Oct 29 23:04:33.914283 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.914258 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-437981-openshift-default-7b4d6f69cc-xlqcw"] Oct 29 23:04:33.933219 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:33.933193 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:34.070935 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:34.070902 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:34.212299 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:34.212271 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:35.566597 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:35.566485 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:35.637591 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:35.637557 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" path="/var/lib/kubelet/pods/8ea5c830-31e9-498e-a774-9c3400e83178/volumes" Oct 29 23:04:35.708260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:35.708229 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:36.549187 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.549152 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 23:04:36.584685 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.584660 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 23:04:36.586164 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.586144 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef0069e-4eb3-4789-a080-b9d4920a5140-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef0069e-4eb3-4789-a080-b9d4920a5140-plugins-config-volume\") pod \"9ef0069e-4eb3-4789-a080-b9d4920a5140\" (UID: \"9ef0069e-4eb3-4789-a080-b9d4920a5140\") " Oct 29 23:04:36.586278 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.586265 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-kube-api-access-cs5qn\" (UniqueName: \"kubernetes.io/projected/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-kube-api-access-cs5qn\") pod \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\" (UID: \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\") " Oct 29 23:04:36.586320 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.586308 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-plugins-config-volume\") pod \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\" (UID: \"eb67a1af-c1e4-4990-8c95-5a9c11f9f625\") " Oct 29 23:04:36.586482 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.586463 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef0069e-4eb3-4789-a080-b9d4920a5140-kube-api-access-57n6z\" (UniqueName: \"kubernetes.io/projected/9ef0069e-4eb3-4789-a080-b9d4920a5140-kube-api-access-57n6z\") pod \"9ef0069e-4eb3-4789-a080-b9d4920a5140\" (UID: \"9ef0069e-4eb3-4789-a080-b9d4920a5140\") " Oct 29 23:04:36.586579 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.586502 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef0069e-4eb3-4789-a080-b9d4920a5140-plugins-config-volume" pod "9ef0069e-4eb3-4789-a080-b9d4920a5140" (UID: "9ef0069e-4eb3-4789-a080-b9d4920a5140"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:36.586753 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.586733 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-plugins-config-volume" pod "eb67a1af-c1e4-4990-8c95-5a9c11f9f625" (UID: "eb67a1af-c1e4-4990-8c95-5a9c11f9f625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:36.587004 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.586986 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef0069e-4eb3-4789-a080-b9d4920a5140-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:36.587071 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.587008 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:36.588418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.588378 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-kube-api-access-cs5qn" pod "eb67a1af-c1e4-4990-8c95-5a9c11f9f625" (UID: "eb67a1af-c1e4-4990-8c95-5a9c11f9f625"). InnerVolumeSpecName "kube-api-access-cs5qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:36.588502 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.588482 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef0069e-4eb3-4789-a080-b9d4920a5140-kube-api-access-57n6z" pod "9ef0069e-4eb3-4789-a080-b9d4920a5140" (UID: "9ef0069e-4eb3-4789-a080-b9d4920a5140"). InnerVolumeSpecName "kube-api-access-57n6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:36.688143 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.688102 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57n6z\" (UniqueName: \"kubernetes.io/projected/9ef0069e-4eb3-4789-a080-b9d4920a5140-kube-api-access-57n6z\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:36.688358 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.688143 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs5qn\" (UniqueName: \"kubernetes.io/projected/eb67a1af-c1e4-4990-8c95-5a9c11f9f625-kube-api-access-cs5qn\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:36.853563 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.853482 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:36.915461 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.915424 5610 generic.go:356] "Generic (PLEG): container finished" podID="9ef0069e-4eb3-4789-a080-b9d4920a5140" containerID="666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f" exitCode=0 Oct 29 23:04:36.915670 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.915485 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" Oct 29 23:04:36.915670 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.915552 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" event={"ID":"9ef0069e-4eb3-4789-a080-b9d4920a5140","Type":"ContainerDied","Data":"666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f"} Oct 29 23:04:36.915670 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.915613 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2" event={"ID":"9ef0069e-4eb3-4789-a080-b9d4920a5140","Type":"ContainerDied","Data":"25cf8e43c8c0905ff2f2f429ec8daa57cd92b0e257ca43f11efe3e77f096e6e3"} Oct 29 23:04:36.915670 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.915645 5610 scope.go:122] "RemoveContainer" containerID="666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f" Oct 29 23:04:36.918920 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.918892 5610 generic.go:356] "Generic (PLEG): container finished" podID="eb67a1af-c1e4-4990-8c95-5a9c11f9f625" containerID="a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645" exitCode=0 Oct 29 23:04:36.919045 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.918947 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" Oct 29 23:04:36.919045 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.918988 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" event={"ID":"eb67a1af-c1e4-4990-8c95-5a9c11f9f625","Type":"ContainerDied","Data":"a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645"} Oct 29 23:04:36.919168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.919042 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv" event={"ID":"eb67a1af-c1e4-4990-8c95-5a9c11f9f625","Type":"ContainerDied","Data":"b5daa9b090a4dc99956782592c13077e4bb180a9257f8431c1a4a921069057e7"} Oct 29 23:04:36.937128 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.931323 5610 scope.go:122] "RemoveContainer" containerID="666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f" Oct 29 23:04:36.937552 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:04:36.937524 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f\": container with ID starting with 666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f not found: ID does not exist" containerID="666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f" Oct 29 23:04:36.937652 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.937561 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f"} err="failed to get container status \"666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f\": rpc error: code = NotFound desc = could not find container \"666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f\": container with ID starting with 666bf3652a66bf2013eeeb0adf704a3130800ec98e7aa094760ac221507a528f not found: ID does not exist" Oct 29 23:04:36.937652 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.937583 5610 scope.go:122] "RemoveContainer" containerID="a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645" Oct 29 23:04:36.949198 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.949175 5610 scope.go:122] "RemoveContainer" containerID="a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645" Oct 29 23:04:36.949570 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:04:36.949548 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645\": container with ID starting with a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645 not found: ID does not exist" containerID="a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645" Oct 29 23:04:36.949658 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.949580 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645"} err="failed to get container status \"a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645\": rpc error: code = NotFound desc = could not find container \"a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645\": container with ID starting with a8a731749126553a8cbcbe5fa6b3acac808f832ce3ee1a370305b52625f90645 not found: ID does not exist" Oct 29 23:04:36.968835 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.968814 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv"] Oct 29 23:04:36.973160 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:36.973139 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-437981-654b69f766-w7klv"] Oct 29 23:04:37.005571 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:37.005549 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2"] Oct 29 23:04:37.025420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:37.012357 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-437981-766d48cb5f-767r2"] Oct 29 23:04:37.637466 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:37.637369 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef0069e-4eb3-4789-a080-b9d4920a5140" path="/var/lib/kubelet/pods/9ef0069e-4eb3-4789-a080-b9d4920a5140/volumes" Oct 29 23:04:37.637849 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:37.637831 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb67a1af-c1e4-4990-8c95-5a9c11f9f625" path="/var/lib/kubelet/pods/eb67a1af-c1e4-4990-8c95-5a9c11f9f625/volumes" Oct 29 23:04:38.168880 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:38.168835 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:38.550781 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:38.550359 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:38.742877 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:38.742834 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" podUID="69efc363-2b9f-48c8-95dc-962e7aade523" containerName="payload-processing" containerID="cri-o://ae1c30834d1b5a566e0284aae08216d105e0f986c348e032cb39e348033724ac" gracePeriod=25 Oct 29 23:04:38.824510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:38.824431 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf"] Oct 29 23:04:38.941430 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:38.941369 5610 generic.go:356] "Generic (PLEG): container finished" podID="69efc363-2b9f-48c8-95dc-962e7aade523" containerID="ae1c30834d1b5a566e0284aae08216d105e0f986c348e032cb39e348033724ac" exitCode=0 Oct 29 23:04:38.941563 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:38.941427 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" event={"ID":"69efc363-2b9f-48c8-95dc-962e7aade523","Type":"ContainerDied","Data":"ae1c30834d1b5a566e0284aae08216d105e0f986c348e032cb39e348033724ac"} Oct 29 23:04:39.015675 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.015639 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:39.110916 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.102789 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:04:39.128537 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.128505 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69efc363-2b9f-48c8-95dc-962e7aade523-kube-api-access-z677g\" (UniqueName: \"kubernetes.io/projected/69efc363-2b9f-48c8-95dc-962e7aade523-kube-api-access-z677g\") pod \"69efc363-2b9f-48c8-95dc-962e7aade523\" (UID: \"69efc363-2b9f-48c8-95dc-962e7aade523\") " Oct 29 23:04:39.128708 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.128602 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69efc363-2b9f-48c8-95dc-962e7aade523-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efc363-2b9f-48c8-95dc-962e7aade523-plugins-config-volume\") pod \"69efc363-2b9f-48c8-95dc-962e7aade523\" (UID: \"69efc363-2b9f-48c8-95dc-962e7aade523\") " Oct 29 23:04:39.128981 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.128953 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69efc363-2b9f-48c8-95dc-962e7aade523-plugins-config-volume" pod "69efc363-2b9f-48c8-95dc-962e7aade523" (UID: "69efc363-2b9f-48c8-95dc-962e7aade523"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:39.129303 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.129284 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efc363-2b9f-48c8-95dc-962e7aade523-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:39.130756 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.130725 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69efc363-2b9f-48c8-95dc-962e7aade523-kube-api-access-z677g" pod "69efc363-2b9f-48c8-95dc-962e7aade523" (UID: "69efc363-2b9f-48c8-95dc-962e7aade523"). InnerVolumeSpecName "kube-api-access-z677g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:39.177851 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.175445 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:39.230794 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.230752 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z677g\" (UniqueName: \"kubernetes.io/projected/69efc363-2b9f-48c8-95dc-962e7aade523-kube-api-access-z677g\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:39.949970 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.949940 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" Oct 29 23:04:39.949970 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.949952 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc" event={"ID":"69efc363-2b9f-48c8-95dc-962e7aade523","Type":"ContainerDied","Data":"55d0f8ca437712c9b0290c330e8c0cbc6f6bd7fd2d1a104584f450e9acea17f0"} Oct 29 23:04:39.950554 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:39.950013 5610 scope.go:122] "RemoveContainer" containerID="ae1c30834d1b5a566e0284aae08216d105e0f986c348e032cb39e348033724ac" Oct 29 23:04:40.004525 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:40.004484 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc"] Oct 29 23:04:40.016104 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:40.016071 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-7b8b55996d-zt9mc"] Oct 29 23:04:40.541858 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:40.541818 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:41.072006 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:41.071968 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:41.637208 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:41.637168 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69efc363-2b9f-48c8-95dc-962e7aade523" path="/var/lib/kubelet/pods/69efc363-2b9f-48c8-95dc-962e7aade523/volumes" Oct 29 23:04:42.434470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:42.434432 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:42.674579 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:42.674544 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:42.825386 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:42.825351 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:42.964329 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:42.958561 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7"] Oct 29 23:04:42.964329 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:42.958824 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" containerID="cri-o://7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d" gracePeriod=30 Oct 29 23:04:43.037324 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:43.037290 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:43.715893 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:04:43.715857 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:04:43.825758 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:43.825713 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" podUID="f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" containerName="payload-processing" containerID="cri-o://7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5" gracePeriod=25 Oct 29 23:04:44.051449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:44.051243 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:45.605902 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:45.605833 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:47.182097 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:47.182031 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:48.239067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.239002 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:04:48.360535 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360499 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-envoy\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.360717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360555 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-token\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-token\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.360717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360583 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-certs\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.360717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360650 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-data\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.360717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360711 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-kube-api-access-4m97k\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-kube-api-access-4m97k\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.360965 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360756 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2d1917-51e2-4f63-ae25-c450d0be21df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2d1917-51e2-4f63-ae25-c450d0be21df-istiod-ca-cert\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.360965 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360830 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-credential-socket\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.360965 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360886 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-socket\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.360965 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.360920 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-podinfo\") pod \"9e2d1917-51e2-4f63-ae25-c450d0be21df\" (UID: \"9e2d1917-51e2-4f63-ae25-c450d0be21df\") " Oct 29 23:04:48.361606 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361271 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-credential-socket" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.361606 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361292 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-certs" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.361606 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361329 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-socket" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.361606 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361408 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2d1917-51e2-4f63-ae25-c450d0be21df-istiod-ca-cert" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:04:48.361864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361669 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.361864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361690 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2d1917-51e2-4f63-ae25-c450d0be21df-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.361864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361800 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-data" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.361864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361705 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.361864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.361848 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.363934 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.363909 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-envoy" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:04:48.364056 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.364003 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-kube-api-access-4m97k" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "kube-api-access-4m97k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:48.364166 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.364140 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-podinfo" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:04:48.364424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.364367 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-token" pod "9e2d1917-51e2-4f63-ae25-c450d0be21df" (UID: "9e2d1917-51e2-4f63-ae25-c450d0be21df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:04:48.463574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.463535 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.463574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.463569 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.463574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.463578 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.463819 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.463587 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m97k\" (UniqueName: \"kubernetes.io/projected/9e2d1917-51e2-4f63-ae25-c450d0be21df-kube-api-access-4m97k\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.463819 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.463597 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2d1917-51e2-4f63-ae25-c450d0be21df-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:04:48.946846 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:48.946808 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:49.020707 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.020669 5610 generic.go:356] "Generic (PLEG): container finished" podID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerID="7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d" exitCode=0 Oct 29 23:04:49.020871 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.020714 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" event={"ID":"9e2d1917-51e2-4f63-ae25-c450d0be21df","Type":"ContainerDied","Data":"7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d"} Oct 29 23:04:49.020871 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.020722 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" Oct 29 23:04:49.020871 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.020756 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7" event={"ID":"9e2d1917-51e2-4f63-ae25-c450d0be21df","Type":"ContainerDied","Data":"080030121e2ef10f93757e6b348e8342a55d5c6d8fa971f27d838f0ac67ef3bc"} Oct 29 23:04:49.020871 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.020776 5610 scope.go:122] "RemoveContainer" containerID="7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d" Oct 29 23:04:49.032332 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.032306 5610 scope.go:122] "RemoveContainer" containerID="7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d" Oct 29 23:04:49.032804 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:04:49.032782 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d\": container with ID starting with 7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d not found: ID does not exist" containerID="7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d" Oct 29 23:04:49.032870 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.032817 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d"} err="failed to get container status \"7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d\": rpc error: code = NotFound desc = could not find container \"7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d\": container with ID starting with 7ad9693e4120e686bda6dfa033876023b95c5ce22c9e88242e4aa4a1402c5e9d not found: ID does not exist" Oct 29 23:04:49.070774 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.070730 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7"] Oct 29 23:04:49.084517 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.084477 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1dabb8-openshift-default-b6d9979bc-4cfl7"] Oct 29 23:04:49.638324 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:49.638288 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" path="/var/lib/kubelet/pods/9e2d1917-51e2-4f63-ae25-c450d0be21df/volumes" Oct 29 23:04:50.140523 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.140494 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:50.285836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.285800 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:50.487099 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.487060 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx"] Oct 29 23:04:50.488145 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488119 5610 cpu_manager.go:414] "containerMap: removing container" podUID="eb67a1af-c1e4-4990-8c95-5a9c11f9f625" containerName="payload-processing" Oct 29 23:04:50.488145 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488146 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb67a1af-c1e4-4990-8c95-5a9c11f9f625" containerName="payload-processing" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488173 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9ef0069e-4eb3-4789-a080-b9d4920a5140" containerName="payload-pre-processing" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488182 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef0069e-4eb3-4789-a080-b9d4920a5140" containerName="payload-pre-processing" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488197 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488205 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488219 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488226 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488239 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488246 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488269 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488278 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488309 5610 cpu_manager.go:414] "containerMap: removing container" podUID="4474e487-1772-417f-82d8-c52932f1b21a" containerName="maas-api" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488316 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4474e487-1772-417f-82d8-c52932f1b21a" containerName="maas-api" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488330 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" Oct 29 23:04:50.488333 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488337 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488370 5610 cpu_manager.go:414] "containerMap: removing container" podUID="69efc363-2b9f-48c8-95dc-962e7aade523" containerName="payload-processing" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488377 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69efc363-2b9f-48c8-95dc-962e7aade523" containerName="payload-processing" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488522 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488539 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="4474e487-1772-417f-82d8-c52932f1b21a" containerName="maas-api" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488551 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2d1917-51e2-4f63-ae25-c450d0be21df" containerName="istio-proxy" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488564 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="69efc363-2b9f-48c8-95dc-962e7aade523" containerName="payload-processing" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488575 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb67a1af-c1e4-4990-8c95-5a9c11f9f625" containerName="payload-processing" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488584 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef0069e-4eb3-4789-a080-b9d4920a5140" containerName="payload-pre-processing" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488592 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" Oct 29 23:04:50.489083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.488602 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea5c830-31e9-498e-a774-9c3400e83178" containerName="istio-proxy" Oct 29 23:04:50.495109 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.495083 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:50.512452 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.512419 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx"] Oct 29 23:04:50.520436 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.519513 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eddb03\"" Oct 29 23:04:50.520436 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.519604 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eddb03-dockercfg-2c5q4\"" Oct 29 23:04:50.591171 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.591137 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3331beb8-c6f0-4036-a335-93ca6648616f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx\" (UID: \"3331beb8-c6f0-4036-a335-93ca6648616f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:50.591387 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.591294 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28t25\" (UniqueName: \"kubernetes.io/projected/3331beb8-c6f0-4036-a335-93ca6648616f-kube-api-access-28t25\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx\" (UID: \"3331beb8-c6f0-4036-a335-93ca6648616f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:50.661378 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.661334 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677"] Oct 29 23:04:50.662605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.662580 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e83482a-0401-47df-ab5a-1b419b3c371d" containerName="istio-proxy" Oct 29 23:04:50.666888 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.666857 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:50.687752 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.687723 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677"] Oct 29 23:04:50.692305 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.692276 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28t25\" (UniqueName: \"kubernetes.io/projected/3331beb8-c6f0-4036-a335-93ca6648616f-kube-api-access-28t25\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx\" (UID: \"3331beb8-c6f0-4036-a335-93ca6648616f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:50.692514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.692478 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3331beb8-c6f0-4036-a335-93ca6648616f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx\" (UID: \"3331beb8-c6f0-4036-a335-93ca6648616f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:50.693182 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.693159 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3331beb8-c6f0-4036-a335-93ca6648616f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx\" (UID: \"3331beb8-c6f0-4036-a335-93ca6648616f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:50.713309 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.713271 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28t25\" (UniqueName: \"kubernetes.io/projected/3331beb8-c6f0-4036-a335-93ca6648616f-kube-api-access-28t25\") pod \"payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx\" (UID: \"3331beb8-c6f0-4036-a335-93ca6648616f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:50.794356 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.794260 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b832c48-65fb-4aa1-b3fe-80e242f6e134-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677\" (UID: \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:50.794628 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.794596 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7xv\" (UniqueName: \"kubernetes.io/projected/3b832c48-65fb-4aa1-b3fe-80e242f6e134-kube-api-access-7c7xv\") pod \"payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677\" (UID: \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:50.811049 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.811012 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:50.903325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.901166 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7xv\" (UniqueName: \"kubernetes.io/projected/3b832c48-65fb-4aa1-b3fe-80e242f6e134-kube-api-access-7c7xv\") pod \"payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677\" (UID: \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:50.903325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.901275 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b832c48-65fb-4aa1-b3fe-80e242f6e134-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677\" (UID: \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:50.903325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.902321 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b832c48-65fb-4aa1-b3fe-80e242f6e134-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677\" (UID: \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:50.921485 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.921452 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7xv\" (UniqueName: \"kubernetes.io/projected/3b832c48-65fb-4aa1-b3fe-80e242f6e134-kube-api-access-7c7xv\") pod \"payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677\" (UID: \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:50.983587 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:50.983553 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:51.123666 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:51.123640 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx"] Oct 29 23:04:51.125351 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:04:51.125324 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3331beb8_c6f0_4036_a335_93ca6648616f.slice/crio-e9aee75bc21c22e6bcbed9aa2103051573b5e4ff08efb10e9e6af80de4d84449 WatchSource:0}: Error finding container e9aee75bc21c22e6bcbed9aa2103051573b5e4ff08efb10e9e6af80de4d84449: Status 404 returned error can't find the container with id e9aee75bc21c22e6bcbed9aa2103051573b5e4ff08efb10e9e6af80de4d84449 Oct 29 23:04:51.286744 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:51.286373 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677"] Oct 29 23:04:51.289246 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:04:51.289212 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b832c48_65fb_4aa1_b3fe_80e242f6e134.slice/crio-72675c93f7c99e4731652083def44b4e889d881293ce916d3a8cb53de0aac949 WatchSource:0}: Error finding container 72675c93f7c99e4731652083def44b4e889d881293ce916d3a8cb53de0aac949: Status 404 returned error can't find the container with id 72675c93f7c99e4731652083def44b4e889d881293ce916d3a8cb53de0aac949 Oct 29 23:04:51.330101 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:51.330055 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:52.052671 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.052623 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" event={"ID":"3331beb8-c6f0-4036-a335-93ca6648616f","Type":"ContainerStarted","Data":"1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632"} Oct 29 23:04:52.053177 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.052734 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" event={"ID":"3331beb8-c6f0-4036-a335-93ca6648616f","Type":"ContainerStarted","Data":"e9aee75bc21c22e6bcbed9aa2103051573b5e4ff08efb10e9e6af80de4d84449"} Oct 29 23:04:52.053544 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.053377 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:04:52.055059 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.055027 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" event={"ID":"3b832c48-65fb-4aa1-b3fe-80e242f6e134","Type":"ContainerStarted","Data":"0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045"} Oct 29 23:04:52.055203 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.055084 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" event={"ID":"3b832c48-65fb-4aa1-b3fe-80e242f6e134","Type":"ContainerStarted","Data":"72675c93f7c99e4731652083def44b4e889d881293ce916d3a8cb53de0aac949"} Oct 29 23:04:52.055368 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.055348 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:04:52.092770 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.092697 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" podStartSLOduration=2.092678559 podStartE2EDuration="2.092678559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:52.090832229 +0000 UTC m=+2954.572149722" watchObservedRunningTime="2026-10-29 23:04:52.092678559 +0000 UTC m=+2954.573996055" Oct 29 23:04:52.131941 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.131871 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" podStartSLOduration=2.131856039 podStartE2EDuration="2.131856039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:52.129606074 +0000 UTC m=+2954.610923567" watchObservedRunningTime="2026-10-29 23:04:52.131856039 +0000 UTC m=+2954.613173532" Oct 29 23:04:52.702227 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.702174 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:52.929465 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:52.929383 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" podUID="f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:04:54.278417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:54.272867 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:55.790440 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:55.790404 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:57.484498 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:57.484461 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:57.920197 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:57.920087 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:58.216144 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:58.216106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:58.368202 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:58.368163 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:58.728293 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:58.728220 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:59.553599 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:59.553559 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:04:59.842467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:04:59.831778 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:00.069838 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.069806 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc"] Oct 29 23:05:00.076115 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.076089 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:00.102630 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.102554 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c74e55\"" Oct 29 23:05:00.102630 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.102568 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c74e55-dockercfg-c47lc\"" Oct 29 23:05:00.108427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.108362 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc"] Oct 29 23:05:00.244772 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.244727 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8rv\" (UniqueName: \"kubernetes.io/projected/65038ec5-df43-48fa-b380-d02bc2ec4a83-kube-api-access-lf8rv\") pod \"payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc\" (UID: \"65038ec5-df43-48fa-b380-d02bc2ec4a83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:00.244986 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.244862 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65038ec5-df43-48fa-b380-d02bc2ec4a83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc\" (UID: \"65038ec5-df43-48fa-b380-d02bc2ec4a83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:00.346078 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.346039 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8rv\" (UniqueName: \"kubernetes.io/projected/65038ec5-df43-48fa-b380-d02bc2ec4a83-kube-api-access-lf8rv\") pod \"payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc\" (UID: \"65038ec5-df43-48fa-b380-d02bc2ec4a83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:00.346287 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.346105 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65038ec5-df43-48fa-b380-d02bc2ec4a83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc\" (UID: \"65038ec5-df43-48fa-b380-d02bc2ec4a83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:00.346907 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.346874 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65038ec5-df43-48fa-b380-d02bc2ec4a83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc\" (UID: \"65038ec5-df43-48fa-b380-d02bc2ec4a83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:00.366387 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.366312 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8rv\" (UniqueName: \"kubernetes.io/projected/65038ec5-df43-48fa-b380-d02bc2ec4a83-kube-api-access-lf8rv\") pod \"payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc\" (UID: \"65038ec5-df43-48fa-b380-d02bc2ec4a83\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:00.395101 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.395068 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:00.692065 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.691961 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:00.696056 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.696029 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc"] Oct 29 23:05:00.710421 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:05:00.697230 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65038ec5_df43_48fa_b380_d02bc2ec4a83.slice/crio-75e1a162f10691c6877a39e01111c2e38f46d89fdd9d60725dc6d4f217ef4c7f WatchSource:0}: Error finding container 75e1a162f10691c6877a39e01111c2e38f46d89fdd9d60725dc6d4f217ef4c7f: Status 404 returned error can't find the container with id 75e1a162f10691c6877a39e01111c2e38f46d89fdd9d60725dc6d4f217ef4c7f Oct 29 23:05:00.832898 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:00.832862 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:01.121860 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:01.121823 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" event={"ID":"65038ec5-df43-48fa-b380-d02bc2ec4a83","Type":"ContainerStarted","Data":"c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009"} Oct 29 23:05:01.122270 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:01.121887 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" event={"ID":"65038ec5-df43-48fa-b380-d02bc2ec4a83","Type":"ContainerStarted","Data":"75e1a162f10691c6877a39e01111c2e38f46d89fdd9d60725dc6d4f217ef4c7f"} Oct 29 23:05:01.122270 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:01.121914 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:01.166034 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:01.165978 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" podStartSLOduration=1.165963346 podStartE2EDuration="1.165963346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:01.16355787 +0000 UTC m=+2963.644875364" watchObservedRunningTime="2026-10-29 23:05:01.165963346 +0000 UTC m=+2963.647280839" Oct 29 23:05:02.162413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:02.161516 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:02.927951 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:02.927914 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" podUID="f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:05:03.064663 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:03.064631 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:05:03.065092 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:03.065077 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:05:03.520416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:03.517588 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:04.120595 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.120562 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 23:05:04.146046 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.146010 5610 generic.go:356] "Generic (PLEG): container finished" podID="f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" containerID="7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5" exitCode=0 Oct 29 23:05:04.146196 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.146053 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" event={"ID":"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4","Type":"ContainerDied","Data":"7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5"} Oct 29 23:05:04.146196 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.146060 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" Oct 29 23:05:04.146196 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.146106 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf" event={"ID":"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4","Type":"ContainerDied","Data":"4fe7a9f6f9a3c59cb187e6a637fef580a02e4c3067b62f7718f2dc5417c5c806"} Oct 29 23:05:04.146196 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.146137 5610 scope.go:122] "RemoveContainer" containerID="7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5" Oct 29 23:05:04.156205 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.156183 5610 scope.go:122] "RemoveContainer" containerID="7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5" Oct 29 23:05:04.156492 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:04.156473 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5\": container with ID starting with 7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5 not found: ID does not exist" containerID="7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5" Oct 29 23:05:04.156547 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.156500 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5"} err="failed to get container status \"7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5\": rpc error: code = NotFound desc = could not find container \"7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5\": container with ID starting with 7f9de61d56f0341d5ddf1be3e8952c1431302c7f5e98ab88be41ad156bf157f5 not found: ID does not exist" Oct 29 23:05:04.201973 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.201927 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-plugins-config-volume\") pod \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\" (UID: \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\") " Oct 29 23:05:04.202804 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.202775 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-plugins-config-volume" pod "f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" (UID: "f05ee9f3-b2e7-49ac-a2d8-a142d73466b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:04.303062 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.303018 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-kube-api-access-cmms7\" (UniqueName: \"kubernetes.io/projected/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-kube-api-access-cmms7\") pod \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\" (UID: \"f05ee9f3-b2e7-49ac-a2d8-a142d73466b4\") " Oct 29 23:05:04.303648 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.303622 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:04.305580 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.305555 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-kube-api-access-cmms7" pod "f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" (UID: "f05ee9f3-b2e7-49ac-a2d8-a142d73466b4"). InnerVolumeSpecName "kube-api-access-cmms7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:04.405092 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.405011 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmms7\" (UniqueName: \"kubernetes.io/projected/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4-kube-api-access-cmms7\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:04.493911 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.493873 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf"] Oct 29 23:05:04.501308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.501239 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-dbb507-7dfdd6d664-xtqpf"] Oct 29 23:05:04.885869 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:04.885835 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:05.046683 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:05.046648 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:05.157518 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:05.157411 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw"] Oct 29 23:05:05.157737 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:05.157711 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" containerID="cri-o://ddfee9d5824929e70f17998ede3702e0e1bb213864b863c143b2e72a934de0cf" gracePeriod=30 Oct 29 23:05:05.229425 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:05.229370 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:05.375603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:05.375564 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:05.637741 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:05.637710 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" path="/var/lib/kubelet/pods/f05ee9f3-b2e7-49ac-a2d8-a142d73466b4/volumes" Oct 29 23:05:05.724910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:05.724847 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:07.130719 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:07.128015 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:07.162475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:07.162442 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:05:07.162663 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:07.162497 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:05:08.592096 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.592052 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx"] Oct 29 23:05:08.592601 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.592343 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" podUID="86dd2ee8-782a-4af7-bc3a-d5afc675e88f" containerName="maas-api" containerID="cri-o://590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f" gracePeriod=30 Oct 29 23:05:08.719272 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.719241 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:08.758353 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.758293 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f"] Oct 29 23:05:08.758617 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.758590 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" podUID="91a0bb3e-f76d-4030-a53c-ac5905587c93" containerName="payload-processing" containerID="cri-o://83716364df0d8911b542caa7e49ee33d89b6032fdd853d8ddc5898af0cef21cf" gracePeriod=30 Oct 29 23:05:08.784818 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.783662 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm"] Oct 29 23:05:08.784818 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.783899 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" podUID="43d156d3-b215-4564-9052-db75adf96095" containerName="payload-pre-processing" containerID="cri-o://8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d" gracePeriod=30 Oct 29 23:05:08.935384 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.931315 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:08.975211 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:08.975176 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.071003 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-maas-api-tls\") pod \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.071638 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-kube-api-access-z7r82\" (UniqueName: \"kubernetes.io/projected/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-kube-api-access-z7r82\") pod \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.072037 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-openshift-service-ca\") pod \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.072458 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-openshift-service-ca" pod "86dd2ee8-782a-4af7-bc3a-d5afc675e88f" (UID: "86dd2ee8-782a-4af7-bc3a-d5afc675e88f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.072597 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-metrics-certs\") pod \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\" (UID: \"86dd2ee8-782a-4af7-bc3a-d5afc675e88f\") " Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.073419 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.076535 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-maas-api-tls" pod "86dd2ee8-782a-4af7-bc3a-d5afc675e88f" (UID: "86dd2ee8-782a-4af7-bc3a-d5afc675e88f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.076894 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-metrics-certs" pod "86dd2ee8-782a-4af7-bc3a-d5afc675e88f" (UID: "86dd2ee8-782a-4af7-bc3a-d5afc675e88f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:09.085415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.077239 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-kube-api-access-z7r82" pod "86dd2ee8-782a-4af7-bc3a-d5afc675e88f" (UID: "86dd2ee8-782a-4af7-bc3a-d5afc675e88f"). InnerVolumeSpecName "kube-api-access-z7r82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:09.101297 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.101184 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:09.113127 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.113105 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:05:09.192424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.180514 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d156d3-b215-4564-9052-db75adf96095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d156d3-b215-4564-9052-db75adf96095-plugins-config-volume\") pod \"43d156d3-b215-4564-9052-db75adf96095\" (UID: \"43d156d3-b215-4564-9052-db75adf96095\") " Oct 29 23:05:09.192424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.180641 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d156d3-b215-4564-9052-db75adf96095-kube-api-access-fj45g\" (UniqueName: \"kubernetes.io/projected/43d156d3-b215-4564-9052-db75adf96095-kube-api-access-fj45g\") pod \"43d156d3-b215-4564-9052-db75adf96095\" (UID: \"43d156d3-b215-4564-9052-db75adf96095\") " Oct 29 23:05:09.192424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.181030 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7r82\" (UniqueName: \"kubernetes.io/projected/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-kube-api-access-z7r82\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.192424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.181049 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.192424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.181061 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86dd2ee8-782a-4af7-bc3a-d5afc675e88f-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.192424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.181824 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d156d3-b215-4564-9052-db75adf96095-plugins-config-volume" pod "43d156d3-b215-4564-9052-db75adf96095" (UID: "43d156d3-b215-4564-9052-db75adf96095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:09.255363 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.255325 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d156d3-b215-4564-9052-db75adf96095-kube-api-access-fj45g" pod "43d156d3-b215-4564-9052-db75adf96095" (UID: "43d156d3-b215-4564-9052-db75adf96095"). InnerVolumeSpecName "kube-api-access-fj45g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:09.258672 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.258575 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:09.263752 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.263546 5610 generic.go:356] "Generic (PLEG): container finished" podID="91a0bb3e-f76d-4030-a53c-ac5905587c93" containerID="83716364df0d8911b542caa7e49ee33d89b6032fdd853d8ddc5898af0cef21cf" exitCode=0 Oct 29 23:05:09.263752 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.263680 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" event={"ID":"91a0bb3e-f76d-4030-a53c-ac5905587c93","Type":"ContainerDied","Data":"83716364df0d8911b542caa7e49ee33d89b6032fdd853d8ddc5898af0cef21cf"} Oct 29 23:05:09.266021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.265541 5610 generic.go:356] "Generic (PLEG): container finished" podID="43d156d3-b215-4564-9052-db75adf96095" containerID="8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d" exitCode=0 Oct 29 23:05:09.266021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.265711 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" event={"ID":"43d156d3-b215-4564-9052-db75adf96095","Type":"ContainerDied","Data":"8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d"} Oct 29 23:05:09.266021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.265754 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" event={"ID":"43d156d3-b215-4564-9052-db75adf96095","Type":"ContainerDied","Data":"4a88dfb44233fc3e8464416dfb96fd4539c93d655ecd337a767c04dc925d7ad0"} Oct 29 23:05:09.266021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.265782 5610 scope.go:122] "RemoveContainer" containerID="8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d" Oct 29 23:05:09.266021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.265926 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm" Oct 29 23:05:09.282681 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.282278 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d156d3-b215-4564-9052-db75adf96095-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.282681 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.282307 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj45g\" (UniqueName: \"kubernetes.io/projected/43d156d3-b215-4564-9052-db75adf96095-kube-api-access-fj45g\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.287309 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.284646 5610 generic.go:356] "Generic (PLEG): container finished" podID="86dd2ee8-782a-4af7-bc3a-d5afc675e88f" containerID="590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f" exitCode=0 Oct 29 23:05:09.287309 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.284822 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" Oct 29 23:05:09.292348 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.292298 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" event={"ID":"86dd2ee8-782a-4af7-bc3a-d5afc675e88f","Type":"ContainerDied","Data":"590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f"} Oct 29 23:05:09.292479 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.292386 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx" event={"ID":"86dd2ee8-782a-4af7-bc3a-d5afc675e88f","Type":"ContainerDied","Data":"165b06e9eb34d64adff9e665661e331f0c27e294387089a14d08ad7794567972"} Oct 29 23:05:09.293032 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.293008 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:05:09.302593 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.297358 5610 scope.go:122] "RemoveContainer" containerID="8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d" Oct 29 23:05:09.302593 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:09.297788 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d\": container with ID starting with 8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d not found: ID does not exist" containerID="8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d" Oct 29 23:05:09.302593 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.297819 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d"} err="failed to get container status \"8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d\": rpc error: code = NotFound desc = could not find container \"8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d\": container with ID starting with 8b24b334471096fc01e211cd051695f0b0106c95fffbc933c2c80b3f7a63414d not found: ID does not exist" Oct 29 23:05:09.302593 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.297835 5610 scope.go:122] "RemoveContainer" containerID="590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f" Oct 29 23:05:09.308977 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.308934 5610 scope.go:122] "RemoveContainer" containerID="590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f" Oct 29 23:05:09.309339 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:09.309284 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f\": container with ID starting with 590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f not found: ID does not exist" containerID="590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f" Oct 29 23:05:09.309339 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.309324 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f"} err="failed to get container status \"590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f\": rpc error: code = NotFound desc = could not find container \"590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f\": container with ID starting with 590dac2f41a4d7740e9c4d9a288effa0947d44f4e9aa29e4fd6911a8c7a1446f not found: ID does not exist" Oct 29 23:05:09.372256 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.372149 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm"] Oct 29 23:05:09.375797 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.375768 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0906f4-6c8b7d59c5-rf9mm"] Oct 29 23:05:09.384173 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.383216 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a0bb3e-f76d-4030-a53c-ac5905587c93-kube-api-access-ct5ph\" (UniqueName: \"kubernetes.io/projected/91a0bb3e-f76d-4030-a53c-ac5905587c93-kube-api-access-ct5ph\") pod \"91a0bb3e-f76d-4030-a53c-ac5905587c93\" (UID: \"91a0bb3e-f76d-4030-a53c-ac5905587c93\") " Oct 29 23:05:09.384173 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.383293 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a0bb3e-f76d-4030-a53c-ac5905587c93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a0bb3e-f76d-4030-a53c-ac5905587c93-plugins-config-volume\") pod \"91a0bb3e-f76d-4030-a53c-ac5905587c93\" (UID: \"91a0bb3e-f76d-4030-a53c-ac5905587c93\") " Oct 29 23:05:09.384716 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.384672 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a0bb3e-f76d-4030-a53c-ac5905587c93-plugins-config-volume" pod "91a0bb3e-f76d-4030-a53c-ac5905587c93" (UID: "91a0bb3e-f76d-4030-a53c-ac5905587c93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:09.386776 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.386749 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a0bb3e-f76d-4030-a53c-ac5905587c93-kube-api-access-ct5ph" pod "91a0bb3e-f76d-4030-a53c-ac5905587c93" (UID: "91a0bb3e-f76d-4030-a53c-ac5905587c93"). InnerVolumeSpecName "kube-api-access-ct5ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:09.420048 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.419115 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:09.449906 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.449871 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx"] Oct 29 23:05:09.455779 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.455750 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0906f4-7447d9fd97-5s2cx"] Oct 29 23:05:09.485649 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.485607 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct5ph\" (UniqueName: \"kubernetes.io/projected/91a0bb3e-f76d-4030-a53c-ac5905587c93-kube-api-access-ct5ph\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.485649 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.485644 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91a0bb3e-f76d-4030-a53c-ac5905587c93-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:09.637328 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.637250 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d156d3-b215-4564-9052-db75adf96095" path="/var/lib/kubelet/pods/43d156d3-b215-4564-9052-db75adf96095/volumes" Oct 29 23:05:09.637800 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.637783 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86dd2ee8-782a-4af7-bc3a-d5afc675e88f" path="/var/lib/kubelet/pods/86dd2ee8-782a-4af7-bc3a-d5afc675e88f/volumes" Oct 29 23:05:09.932152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:09.932074 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:10.294952 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.294902 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" event={"ID":"91a0bb3e-f76d-4030-a53c-ac5905587c93","Type":"ContainerDied","Data":"0aba4fb966df747c3d2e24e7c0d999cb3f6638e3ec5678cc0fd2c01895c8c5bf"} Oct 29 23:05:10.295148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.294972 5610 scope.go:122] "RemoveContainer" containerID="83716364df0d8911b542caa7e49ee33d89b6032fdd853d8ddc5898af0cef21cf" Oct 29 23:05:10.295148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.294923 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f" Oct 29 23:05:10.299348 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.299319 5610 generic.go:356] "Generic (PLEG): container finished" podID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerID="ddfee9d5824929e70f17998ede3702e0e1bb213864b863c143b2e72a934de0cf" exitCode=0 Oct 29 23:05:10.299529 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.299376 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" event={"ID":"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685","Type":"ContainerDied","Data":"ddfee9d5824929e70f17998ede3702e0e1bb213864b863c143b2e72a934de0cf"} Oct 29 23:05:10.307263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.307242 5610 scope.go:122] "RemoveContainer" containerID="054f76c1d32023c5b85579c7f24d44490023beec38f08a788c78e17a0bca6279" Oct 29 23:05:10.348476 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.348447 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f"] Oct 29 23:05:10.359880 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.359851 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0906f4-7d7f495d74-qf86f"] Oct 29 23:05:10.661876 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.661782 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz"] Oct 29 23:05:10.790053 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.790017 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:10.864585 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.864557 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 23:05:10.902999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.902957 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-envoy\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903162 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903010 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-podinfo\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903162 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903079 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istiod-ca-cert\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903169 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-token\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903238 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-certs\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903372 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903343 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-credential-socket\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903436 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-kube-api-access-77qp7\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-kube-api-access-77qp7\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903552 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903481 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-socket\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903552 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903541 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-data\") pod \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\" (UID: \"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685\") " Oct 29 23:05:10.903672 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903643 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-certs" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.903745 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903667 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istiod-ca-cert" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:10.903910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903884 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-socket" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.903910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903901 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-data" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.904069 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.903915 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-credential-socket" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.904141 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.904127 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.904193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.904146 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.904193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.904159 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.904193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.904173 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.904193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.904185 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:10.905558 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.905535 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-podinfo" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:10.905690 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.905669 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-token" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:10.906120 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.906095 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-envoy" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:10.906212 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:10.906182 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-kube-api-access-77qp7" pod "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" (UID: "ac2d8c3e-e6b1-47a5-a26d-1d923d92a685"). InnerVolumeSpecName "kube-api-access-77qp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:11.005765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.005730 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77qp7\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-kube-api-access-77qp7\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.005765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.005764 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.005765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.005774 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.006016 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.005784 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:11.106775 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.106736 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:11.308861 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.308829 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" Oct 29 23:05:11.308861 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.308856 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw" event={"ID":"ac2d8c3e-e6b1-47a5-a26d-1d923d92a685","Type":"ContainerDied","Data":"c2bedddc9f8cb2122cb614176ea0ebb17043ef941312ea7f7a6fe3480c5c70cc"} Oct 29 23:05:11.309146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.308915 5610 scope.go:122] "RemoveContainer" containerID="ddfee9d5824929e70f17998ede3702e0e1bb213864b863c143b2e72a934de0cf" Oct 29 23:05:11.360226 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.360192 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw"] Oct 29 23:05:11.372939 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.372908 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-dbb507-openshift-default-7bf544d8cf-l6nmw"] Oct 29 23:05:11.637326 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.637237 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a0bb3e-f76d-4030-a53c-ac5905587c93" path="/var/lib/kubelet/pods/91a0bb3e-f76d-4030-a53c-ac5905587c93/volumes" Oct 29 23:05:11.637863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:11.637838 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" path="/var/lib/kubelet/pods/ac2d8c3e-e6b1-47a5-a26d-1d923d92a685/volumes" Oct 29 23:05:12.131604 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:12.131560 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:12.404218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:12.403253 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:13.714167 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:13.714126 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:05:13.874085 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:13.873813 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:15.571670 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:15.571633 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:15.922724 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:15.922624 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:16.080759 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:16.080725 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:16.261223 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:16.261177 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:16.534871 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:16.534769 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:16.681268 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:16.681229 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:17.752845 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:17.751185 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:19.029140 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:19.029097 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6"] Oct 29 23:05:19.040427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:19.040360 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b858176-c85h6"] Oct 29 23:05:19.477205 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:19.477171 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:19.637100 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:19.637064 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37858cc2-c0c8-4d60-b15c-53f65745d14e" path="/var/lib/kubelet/pods/37858cc2-c0c8-4d60-b15c-53f65745d14e/volumes" Oct 29 23:05:20.949762 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:20.949725 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:22.007004 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.006969 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2"] Oct 29 23:05:22.007541 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.007171 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" podUID="3a68a161-97bd-429d-b6bb-28786285b6a9" containerName="maas-api" containerID="cri-o://812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e" gracePeriod=30 Oct 29 23:05:22.116828 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.116796 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:22.148083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.148040 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l"] Oct 29 23:05:22.148308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.148282 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" podUID="56496049-8920-4b5f-b18d-7d536f38feb0" containerName="payload-processing" containerID="cri-o://d246eb1f7eefb4c54cb49940f044ceef2ab0ca40b616384a1ecc6d1ae6b65f95" gracePeriod=30 Oct 29 23:05:22.167310 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.167277 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789"] Oct 29 23:05:22.167618 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.167563 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" podUID="ce072c8e-2cca-4616-916d-2d2ddb52d113" containerName="payload-pre-processing" containerID="cri-o://1d789adba7079dd9d42cea1e90f4de278b43b35be115d5c6e70e5ca20ed60325" gracePeriod=30 Oct 29 23:05:22.272079 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.272027 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:22.384575 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.381602 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:05:22.417413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.416030 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:22.433417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.431821 5610 generic.go:356] "Generic (PLEG): container finished" podID="ce072c8e-2cca-4616-916d-2d2ddb52d113" containerID="1d789adba7079dd9d42cea1e90f4de278b43b35be115d5c6e70e5ca20ed60325" exitCode=0 Oct 29 23:05:22.433417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.431982 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" event={"ID":"ce072c8e-2cca-4616-916d-2d2ddb52d113","Type":"ContainerDied","Data":"1d789adba7079dd9d42cea1e90f4de278b43b35be115d5c6e70e5ca20ed60325"} Oct 29 23:05:22.435669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.434515 5610 generic.go:356] "Generic (PLEG): container finished" podID="3a68a161-97bd-429d-b6bb-28786285b6a9" containerID="812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e" exitCode=0 Oct 29 23:05:22.435669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.434708 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" event={"ID":"3a68a161-97bd-429d-b6bb-28786285b6a9","Type":"ContainerDied","Data":"812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e"} Oct 29 23:05:22.435669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.434759 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" event={"ID":"3a68a161-97bd-429d-b6bb-28786285b6a9","Type":"ContainerDied","Data":"dacc3b9c9ac3c8bd99c306ac0a3b2a7906ffbf6a76004899dc78a207074654cd"} Oct 29 23:05:22.435669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.434791 5610 scope.go:122] "RemoveContainer" containerID="812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e" Oct 29 23:05:22.435669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.434967 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2" Oct 29 23:05:22.440992 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.440429 5610 generic.go:356] "Generic (PLEG): container finished" podID="56496049-8920-4b5f-b18d-7d536f38feb0" containerID="d246eb1f7eefb4c54cb49940f044ceef2ab0ca40b616384a1ecc6d1ae6b65f95" exitCode=0 Oct 29 23:05:22.440992 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.440689 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" event={"ID":"56496049-8920-4b5f-b18d-7d536f38feb0","Type":"ContainerDied","Data":"d246eb1f7eefb4c54cb49940f044ceef2ab0ca40b616384a1ecc6d1ae6b65f95"} Oct 29 23:05:22.519113 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.517956 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a68a161-97bd-429d-b6bb-28786285b6a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a68a161-97bd-429d-b6bb-28786285b6a9-openshift-service-ca\") pod \"3a68a161-97bd-429d-b6bb-28786285b6a9\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " Oct 29 23:05:22.519113 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.518096 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-metrics-certs\") pod \"3a68a161-97bd-429d-b6bb-28786285b6a9\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " Oct 29 23:05:22.519113 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.518260 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a68a161-97bd-429d-b6bb-28786285b6a9-kube-api-access-dtxcf\" (UniqueName: \"kubernetes.io/projected/3a68a161-97bd-429d-b6bb-28786285b6a9-kube-api-access-dtxcf\") pod \"3a68a161-97bd-429d-b6bb-28786285b6a9\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " Oct 29 23:05:22.519113 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.518320 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-maas-api-tls\") pod \"3a68a161-97bd-429d-b6bb-28786285b6a9\" (UID: \"3a68a161-97bd-429d-b6bb-28786285b6a9\") " Oct 29 23:05:22.526609 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.526504 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a68a161-97bd-429d-b6bb-28786285b6a9-openshift-service-ca" pod "3a68a161-97bd-429d-b6bb-28786285b6a9" (UID: "3a68a161-97bd-429d-b6bb-28786285b6a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:22.529097 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.528974 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a68a161-97bd-429d-b6bb-28786285b6a9-kube-api-access-dtxcf" pod "3a68a161-97bd-429d-b6bb-28786285b6a9" (UID: "3a68a161-97bd-429d-b6bb-28786285b6a9"). InnerVolumeSpecName "kube-api-access-dtxcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:22.531764 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.531736 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-maas-api-tls" pod "3a68a161-97bd-429d-b6bb-28786285b6a9" (UID: "3a68a161-97bd-429d-b6bb-28786285b6a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:22.534199 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.534165 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-metrics-certs" pod "3a68a161-97bd-429d-b6bb-28786285b6a9" (UID: "3a68a161-97bd-429d-b6bb-28786285b6a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:22.545509 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.544699 5610 scope.go:122] "RemoveContainer" containerID="812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e" Oct 29 23:05:22.549224 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:22.546286 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e\": container with ID starting with 812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e not found: ID does not exist" containerID="812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e" Oct 29 23:05:22.549224 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.546356 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e"} err="failed to get container status \"812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e\": rpc error: code = NotFound desc = could not find container \"812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e\": container with ID starting with 812d34538ac53042ff46c0d0c9139dd2a24d5aebee19fab30b4547792a2ea48e not found: ID does not exist" Oct 29 23:05:22.560311 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.560284 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:05:22.620240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.619953 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a68a161-97bd-429d-b6bb-28786285b6a9-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.620240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.619983 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.620240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.619995 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtxcf\" (UniqueName: \"kubernetes.io/projected/3a68a161-97bd-429d-b6bb-28786285b6a9-kube-api-access-dtxcf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.620240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.620004 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a68a161-97bd-429d-b6bb-28786285b6a9-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.642379 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.642353 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:05:22.721234 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.721194 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56496049-8920-4b5f-b18d-7d536f38feb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56496049-8920-4b5f-b18d-7d536f38feb0-plugins-config-volume\") pod \"56496049-8920-4b5f-b18d-7d536f38feb0\" (UID: \"56496049-8920-4b5f-b18d-7d536f38feb0\") " Oct 29 23:05:22.721447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.721301 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce072c8e-2cca-4616-916d-2d2ddb52d113-kube-api-access-gjmg2\" (UniqueName: \"kubernetes.io/projected/ce072c8e-2cca-4616-916d-2d2ddb52d113-kube-api-access-gjmg2\") pod \"ce072c8e-2cca-4616-916d-2d2ddb52d113\" (UID: \"ce072c8e-2cca-4616-916d-2d2ddb52d113\") " Oct 29 23:05:22.721494 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.721462 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56496049-8920-4b5f-b18d-7d536f38feb0-kube-api-access-td25g\" (UniqueName: \"kubernetes.io/projected/56496049-8920-4b5f-b18d-7d536f38feb0-kube-api-access-td25g\") pod \"56496049-8920-4b5f-b18d-7d536f38feb0\" (UID: \"56496049-8920-4b5f-b18d-7d536f38feb0\") " Oct 29 23:05:22.721546 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.721528 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce072c8e-2cca-4616-916d-2d2ddb52d113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce072c8e-2cca-4616-916d-2d2ddb52d113-plugins-config-volume\") pod \"ce072c8e-2cca-4616-916d-2d2ddb52d113\" (UID: \"ce072c8e-2cca-4616-916d-2d2ddb52d113\") " Oct 29 23:05:22.722439 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.722412 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce072c8e-2cca-4616-916d-2d2ddb52d113-plugins-config-volume" pod "ce072c8e-2cca-4616-916d-2d2ddb52d113" (UID: "ce072c8e-2cca-4616-916d-2d2ddb52d113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:22.722894 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.722871 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56496049-8920-4b5f-b18d-7d536f38feb0-plugins-config-volume" pod "56496049-8920-4b5f-b18d-7d536f38feb0" (UID: "56496049-8920-4b5f-b18d-7d536f38feb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:22.725177 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.725147 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce072c8e-2cca-4616-916d-2d2ddb52d113-kube-api-access-gjmg2" pod "ce072c8e-2cca-4616-916d-2d2ddb52d113" (UID: "ce072c8e-2cca-4616-916d-2d2ddb52d113"). InnerVolumeSpecName "kube-api-access-gjmg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:22.725773 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.725747 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56496049-8920-4b5f-b18d-7d536f38feb0-kube-api-access-td25g" pod "56496049-8920-4b5f-b18d-7d536f38feb0" (UID: "56496049-8920-4b5f-b18d-7d536f38feb0"). InnerVolumeSpecName "kube-api-access-td25g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:22.791566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.791434 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2"] Oct 29 23:05:22.796331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.794915 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ee1fc-b656468f8-b82k2"] Oct 29 23:05:22.799863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.799837 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:22.823323 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.823284 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56496049-8920-4b5f-b18d-7d536f38feb0-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.823323 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.823317 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjmg2\" (UniqueName: \"kubernetes.io/projected/ce072c8e-2cca-4616-916d-2d2ddb52d113-kube-api-access-gjmg2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.823557 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.823331 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td25g\" (UniqueName: \"kubernetes.io/projected/56496049-8920-4b5f-b18d-7d536f38feb0-kube-api-access-td25g\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.823557 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.823343 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce072c8e-2cca-4616-916d-2d2ddb52d113-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:22.936138 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:22.936099 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:23.147551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.147463 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:23.424051 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.423963 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:23.450144 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.450110 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" Oct 29 23:05:23.450144 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.450120 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l" event={"ID":"56496049-8920-4b5f-b18d-7d536f38feb0","Type":"ContainerDied","Data":"69067fa4a662347c7dc655e658cd98cd4fb2b2538227cc900ff5b291eeba5995"} Oct 29 23:05:23.450434 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.450216 5610 scope.go:122] "RemoveContainer" containerID="d246eb1f7eefb4c54cb49940f044ceef2ab0ca40b616384a1ecc6d1ae6b65f95" Oct 29 23:05:23.452080 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.452058 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" event={"ID":"ce072c8e-2cca-4616-916d-2d2ddb52d113","Type":"ContainerDied","Data":"fe27c4b85d49da6f90397e51f93727f077ec8a179fe2a633e84b95a9a121aad8"} Oct 29 23:05:23.452080 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.452081 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789" Oct 29 23:05:23.464549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.464527 5610 scope.go:122] "RemoveContainer" containerID="1d789adba7079dd9d42cea1e90f4de278b43b35be115d5c6e70e5ca20ed60325" Oct 29 23:05:23.518445 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.518387 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l"] Oct 29 23:05:23.528535 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.528500 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ee1fc-54c499f956-hqp5l"] Oct 29 23:05:23.553883 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.553851 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789"] Oct 29 23:05:23.558314 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.558288 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9ee1fc-574466b74c-vt789"] Oct 29 23:05:23.638294 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.638255 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a68a161-97bd-429d-b6bb-28786285b6a9" path="/var/lib/kubelet/pods/3a68a161-97bd-429d-b6bb-28786285b6a9/volumes" Oct 29 23:05:23.638926 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.638891 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56496049-8920-4b5f-b18d-7d536f38feb0" path="/var/lib/kubelet/pods/56496049-8920-4b5f-b18d-7d536f38feb0/volumes" Oct 29 23:05:23.639313 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:23.639281 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce072c8e-2cca-4616-916d-2d2ddb52d113" path="/var/lib/kubelet/pods/ce072c8e-2cca-4616-916d-2d2ddb52d113/volumes" Oct 29 23:05:25.117223 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:25.117166 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:25.683728 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:25.683689 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" containerID="cri-o://1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac" gracePeriod=45 Oct 29 23:05:25.689447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:25.689388 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Oct 29 23:05:26.069436 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.069382 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 23:05:26.164838 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.164805 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-model-cache\") pod \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " Oct 29 23:05:26.165331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.165062 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tmp-dir\") pod \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " Oct 29 23:05:26.165331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.165167 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-model-cache" pod "6f0a67a0-a252-4a00-94c5-cc659b9a877b" (UID: "6f0a67a0-a252-4a00-94c5-cc659b9a877b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.165331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.165194 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-dshm\") pod \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " Oct 29 23:05:26.165331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.165241 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0a67a0-a252-4a00-94c5-cc659b9a877b-kube-api-access-qhtcg\" (UniqueName: \"kubernetes.io/projected/6f0a67a0-a252-4a00-94c5-cc659b9a877b-kube-api-access-qhtcg\") pod \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " Oct 29 23:05:26.165331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.165276 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tls-certs\") pod \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " Oct 29 23:05:26.165331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.165297 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tmp-dir" pod "6f0a67a0-a252-4a00-94c5-cc659b9a877b" (UID: "6f0a67a0-a252-4a00-94c5-cc659b9a877b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.165734 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.165340 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-home\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-home\") pod \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\" (UID: \"6f0a67a0-a252-4a00-94c5-cc659b9a877b\") " Oct 29 23:05:26.166035 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.166010 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.166124 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.166063 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.166379 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.166356 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-home" pod "6f0a67a0-a252-4a00-94c5-cc659b9a877b" (UID: "6f0a67a0-a252-4a00-94c5-cc659b9a877b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.169569 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.168219 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0a67a0-a252-4a00-94c5-cc659b9a877b-kube-api-access-qhtcg" pod "6f0a67a0-a252-4a00-94c5-cc659b9a877b" (UID: "6f0a67a0-a252-4a00-94c5-cc659b9a877b"). InnerVolumeSpecName "kube-api-access-qhtcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:26.169569 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.168923 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tls-certs" pod "6f0a67a0-a252-4a00-94c5-cc659b9a877b" (UID: "6f0a67a0-a252-4a00-94c5-cc659b9a877b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:26.171927 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.170892 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-dshm" pod "6f0a67a0-a252-4a00-94c5-cc659b9a877b" (UID: "6f0a67a0-a252-4a00-94c5-cc659b9a877b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:26.266976 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.266944 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.266976 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.266977 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhtcg\" (UniqueName: \"kubernetes.io/projected/6f0a67a0-a252-4a00-94c5-cc659b9a877b-kube-api-access-qhtcg\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.267167 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.266991 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f0a67a0-a252-4a00-94c5-cc659b9a877b-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.267167 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.267003 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f0a67a0-a252-4a00-94c5-cc659b9a877b-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:26.476485 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.476443 5610 generic.go:356] "Generic (PLEG): container finished" podID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerID="1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac" exitCode=0 Oct 29 23:05:26.476665 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.476485 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" event={"ID":"6f0a67a0-a252-4a00-94c5-cc659b9a877b","Type":"ContainerDied","Data":"1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac"} Oct 29 23:05:26.476665 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.476532 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" event={"ID":"6f0a67a0-a252-4a00-94c5-cc659b9a877b","Type":"ContainerDied","Data":"c2269e9c37bef97438fad47396d73d166c06ef3f11d5115f5cb02481a99ee73c"} Oct 29 23:05:26.476665 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.476460 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz" Oct 29 23:05:26.476665 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.476554 5610 scope.go:122] "RemoveContainer" containerID="1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac" Oct 29 23:05:26.487263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.487242 5610 scope.go:122] "RemoveContainer" containerID="1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac" Oct 29 23:05:26.487570 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:26.487547 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac\": container with ID starting with 1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac not found: ID does not exist" containerID="1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac" Oct 29 23:05:26.487651 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.487583 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac"} err="failed to get container status \"1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac\": rpc error: code = NotFound desc = could not find container \"1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac\": container with ID starting with 1873323c0e4ba0e97f55f24f988cdf9b39308977345c65d3aefffe1cc6f849ac not found: ID does not exist" Oct 29 23:05:26.527456 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.527373 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz"] Oct 29 23:05:26.534822 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.534788 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-0d12c5/premium-w5-0d12c5-kserve-5dc846f5fb-czndz"] Oct 29 23:05:26.632741 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:26.632695 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:27.163410 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.163365 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m"] Oct 29 23:05:27.209075 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.209044 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6"] Oct 29 23:05:27.369826 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.369792 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:27.517916 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.517882 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:27.637270 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.637232 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" path="/var/lib/kubelet/pods/6f0a67a0-a252-4a00-94c5-cc659b9a877b/volumes" Oct 29 23:05:27.682617 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.682579 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:27.843467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.834429 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j"] Oct 29 23:05:27.844346 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.844297 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" podUID="80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" containerName="maas-api" containerID="cri-o://6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f" gracePeriod=30 Oct 29 23:05:27.942959 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:27.942922 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:28.191215 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.191187 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt"] Oct 29 23:05:28.191470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.191436 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" podUID="caeab242-8754-4bee-8f24-41b26ed86f2d" containerName="maas-api" containerID="cri-o://364758c30f75d090998d2eb9975ef65d1b09f48709fb64e5569389e2f6c93c38" gracePeriod=30 Oct 29 23:05:28.191761 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.191652 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 23:05:28.298344 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.298312 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-openshift-service-ca\") pod \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " Oct 29 23:05:28.298969 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.298429 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-maas-api-tls\") pod \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " Oct 29 23:05:28.298969 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.298593 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-kube-api-access-p8rpj\" (UniqueName: \"kubernetes.io/projected/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-kube-api-access-p8rpj\") pod \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " Oct 29 23:05:28.298969 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.298799 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-openshift-service-ca" pod "80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" (UID: "80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:28.299387 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.299268 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-metrics-certs\") pod \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\" (UID: \"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a\") " Oct 29 23:05:28.299820 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.299800 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.310868 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.307925 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-metrics-certs" pod "80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" (UID: "80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:28.310868 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.308715 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-maas-api-tls" pod "80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" (UID: "80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:28.310868 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.310245 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:28.311664 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.311460 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-kube-api-access-p8rpj" pod "80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" (UID: "80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a"). InnerVolumeSpecName "kube-api-access-p8rpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:28.385650 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.385574 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd"] Oct 29 23:05:28.386669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.386283 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" podUID="460af1df-6d95-4d73-923a-61bd9ba7f11a" containerName="payload-pre-processing" containerID="cri-o://c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a" gracePeriod=30 Oct 29 23:05:28.401698 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.400962 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.401698 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.400993 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rpj\" (UniqueName: \"kubernetes.io/projected/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-kube-api-access-p8rpj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.401698 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.401008 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.490059 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.490022 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:28.497222 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.495939 5610 generic.go:356] "Generic (PLEG): container finished" podID="80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" containerID="6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f" exitCode=0 Oct 29 23:05:28.497222 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.496079 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" Oct 29 23:05:28.497222 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.496102 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" event={"ID":"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a","Type":"ContainerDied","Data":"6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f"} Oct 29 23:05:28.497222 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.497163 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j" event={"ID":"80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a","Type":"ContainerDied","Data":"147e78006a8499f415ef8a4e4be9eaa0dc784d0ed2abc63cae9ca30cfe73d85f"} Oct 29 23:05:28.497222 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.497202 5610 scope.go:122] "RemoveContainer" containerID="6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f" Oct 29 23:05:28.500559 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.500526 5610 generic.go:356] "Generic (PLEG): container finished" podID="caeab242-8754-4bee-8f24-41b26ed86f2d" containerID="364758c30f75d090998d2eb9975ef65d1b09f48709fb64e5569389e2f6c93c38" exitCode=0 Oct 29 23:05:28.501228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.500752 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" event={"ID":"caeab242-8754-4bee-8f24-41b26ed86f2d","Type":"ContainerDied","Data":"364758c30f75d090998d2eb9975ef65d1b09f48709fb64e5569389e2f6c93c38"} Oct 29 23:05:28.516115 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.516087 5610 scope.go:122] "RemoveContainer" containerID="6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f" Oct 29 23:05:28.516550 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:28.516478 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f\": container with ID starting with 6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f not found: ID does not exist" containerID="6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f" Oct 29 23:05:28.516550 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.516521 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f"} err="failed to get container status \"6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f\": rpc error: code = NotFound desc = could not find container \"6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f\": container with ID starting with 6c8a08e6f228f46991033f08f86e6df945ba4e6cb4dc2e5d01d0de4ab704181f not found: ID does not exist" Oct 29 23:05:28.529614 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.529411 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:05:28.547576 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.547491 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j"] Oct 29 23:05:28.556166 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.556133 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-e81b56-7b576564b-vsr6j"] Oct 29 23:05:28.607146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.606896 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-maas-api-tls\") pod \"caeab242-8754-4bee-8f24-41b26ed86f2d\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " Oct 29 23:05:28.607146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.607106 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caeab242-8754-4bee-8f24-41b26ed86f2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeab242-8754-4bee-8f24-41b26ed86f2d-openshift-service-ca\") pod \"caeab242-8754-4bee-8f24-41b26ed86f2d\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " Oct 29 23:05:28.607146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.607141 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-metrics-certs\") pod \"caeab242-8754-4bee-8f24-41b26ed86f2d\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " Oct 29 23:05:28.607484 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.607183 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caeab242-8754-4bee-8f24-41b26ed86f2d-kube-api-access-ln7jj\" (UniqueName: \"kubernetes.io/projected/caeab242-8754-4bee-8f24-41b26ed86f2d-kube-api-access-ln7jj\") pod \"caeab242-8754-4bee-8f24-41b26ed86f2d\" (UID: \"caeab242-8754-4bee-8f24-41b26ed86f2d\") " Oct 29 23:05:28.609142 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.609108 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caeab242-8754-4bee-8f24-41b26ed86f2d-openshift-service-ca" pod "caeab242-8754-4bee-8f24-41b26ed86f2d" (UID: "caeab242-8754-4bee-8f24-41b26ed86f2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:28.625441 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.625347 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-metrics-certs" pod "caeab242-8754-4bee-8f24-41b26ed86f2d" (UID: "caeab242-8754-4bee-8f24-41b26ed86f2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:28.625441 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.625374 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caeab242-8754-4bee-8f24-41b26ed86f2d-kube-api-access-ln7jj" pod "caeab242-8754-4bee-8f24-41b26ed86f2d" (UID: "caeab242-8754-4bee-8f24-41b26ed86f2d"). InnerVolumeSpecName "kube-api-access-ln7jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:28.628959 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.628916 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-maas-api-tls" pod "caeab242-8754-4bee-8f24-41b26ed86f2d" (UID: "caeab242-8754-4bee-8f24-41b26ed86f2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:28.643504 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.643016 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:28.674773 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.674743 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:05:28.709692 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.709645 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.710374 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.709922 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeab242-8754-4bee-8f24-41b26ed86f2d-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.710374 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.709950 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeab242-8754-4bee-8f24-41b26ed86f2d-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.710374 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.709968 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln7jj\" (UniqueName: \"kubernetes.io/projected/caeab242-8754-4bee-8f24-41b26ed86f2d-kube-api-access-ln7jj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.822415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.814320 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460af1df-6d95-4d73-923a-61bd9ba7f11a-kube-api-access-bhg4q\" (UniqueName: \"kubernetes.io/projected/460af1df-6d95-4d73-923a-61bd9ba7f11a-kube-api-access-bhg4q\") pod \"460af1df-6d95-4d73-923a-61bd9ba7f11a\" (UID: \"460af1df-6d95-4d73-923a-61bd9ba7f11a\") " Oct 29 23:05:28.822415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.815113 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460af1df-6d95-4d73-923a-61bd9ba7f11a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460af1df-6d95-4d73-923a-61bd9ba7f11a-plugins-config-volume\") pod \"460af1df-6d95-4d73-923a-61bd9ba7f11a\" (UID: \"460af1df-6d95-4d73-923a-61bd9ba7f11a\") " Oct 29 23:05:28.822415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.816049 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460af1df-6d95-4d73-923a-61bd9ba7f11a-plugins-config-volume" pod "460af1df-6d95-4d73-923a-61bd9ba7f11a" (UID: "460af1df-6d95-4d73-923a-61bd9ba7f11a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:28.842119 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.829885 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:28.842119 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.830520 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460af1df-6d95-4d73-923a-61bd9ba7f11a-kube-api-access-bhg4q" pod "460af1df-6d95-4d73-923a-61bd9ba7f11a" (UID: "460af1df-6d95-4d73-923a-61bd9ba7f11a"). InnerVolumeSpecName "kube-api-access-bhg4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:28.917292 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.917197 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhg4q\" (UniqueName: \"kubernetes.io/projected/460af1df-6d95-4d73-923a-61bd9ba7f11a-kube-api-access-bhg4q\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.917292 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.917227 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460af1df-6d95-4d73-923a-61bd9ba7f11a-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:28.979743 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:28.979698 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:29.135816 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.135751 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:29.501679 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.501646 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:29.509715 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.509684 5610 generic.go:356] "Generic (PLEG): container finished" podID="460af1df-6d95-4d73-923a-61bd9ba7f11a" containerID="c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a" exitCode=0 Oct 29 23:05:29.509870 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.509734 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" Oct 29 23:05:29.509870 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.509764 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" event={"ID":"460af1df-6d95-4d73-923a-61bd9ba7f11a","Type":"ContainerDied","Data":"c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a"} Oct 29 23:05:29.509870 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.509826 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd" event={"ID":"460af1df-6d95-4d73-923a-61bd9ba7f11a","Type":"ContainerDied","Data":"d38eb1aea6e2b9fb9ebc918399a2e82a95afd6d7e3097c569a55b634cf2df455"} Oct 29 23:05:29.509870 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.509858 5610 scope.go:122] "RemoveContainer" containerID="c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a" Oct 29 23:05:29.512971 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.512715 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" Oct 29 23:05:29.512971 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.512732 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt" event={"ID":"caeab242-8754-4bee-8f24-41b26ed86f2d","Type":"ContainerDied","Data":"8c2d3911dc6ef919dafb3fc4948131da7c83d924059459c62a0fe489e4a39576"} Oct 29 23:05:29.523153 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.523047 5610 scope.go:122] "RemoveContainer" containerID="c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a" Oct 29 23:05:29.523493 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:29.523468 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a\": container with ID starting with c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a not found: ID does not exist" containerID="c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a" Oct 29 23:05:29.523610 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.523506 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a"} err="failed to get container status \"c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a\": rpc error: code = NotFound desc = could not find container \"c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a\": container with ID starting with c0c8a95d8f3796e947d1cd9f3a3c25dfb5c29b3afde69b5529d39d8abf8a833a not found: ID does not exist" Oct 29 23:05:29.523610 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.523526 5610 scope.go:122] "RemoveContainer" containerID="364758c30f75d090998d2eb9975ef65d1b09f48709fb64e5569389e2f6c93c38" Oct 29 23:05:29.563045 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.563009 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt"] Oct 29 23:05:29.574438 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.574382 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3e2b2c-6fb66948d6-cx7jt"] Oct 29 23:05:29.614246 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.614200 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd"] Oct 29 23:05:29.617463 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.617425 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3e2b2c-6f556cbd55-b4tfd"] Oct 29 23:05:29.635912 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.635881 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460af1df-6d95-4d73-923a-61bd9ba7f11a" path="/var/lib/kubelet/pods/460af1df-6d95-4d73-923a-61bd9ba7f11a/volumes" Oct 29 23:05:29.636356 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.636340 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" path="/var/lib/kubelet/pods/80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a/volumes" Oct 29 23:05:29.636749 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:29.636736 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caeab242-8754-4bee-8f24-41b26ed86f2d" path="/var/lib/kubelet/pods/caeab242-8754-4bee-8f24-41b26ed86f2d/volumes" Oct 29 23:05:30.259753 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:30.259714 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:30.413060 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:30.413024 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:30.535466 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:30.535371 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw"] Oct 29 23:05:30.535917 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:30.535612 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" containerID="cri-o://a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def" gracePeriod=30 Oct 29 23:05:30.629196 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:30.629166 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:30.771202 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:30.771082 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:32.163900 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:32.163853 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" podUID="82fb649c-6c0b-456c-ae9b-c3e52195fc53" containerName="payload-processing" containerID="cri-o://c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45" gracePeriod=25 Oct 29 23:05:32.209704 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:32.209662 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" podUID="3a59f3f1-c569-4d63-9896-42d459c6f7bd" containerName="payload-pre-processing" containerID="cri-o://83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976" gracePeriod=25 Oct 29 23:05:32.325418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:32.322086 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:33.745231 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:33.745154 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:35.245723 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.245682 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:35.726274 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.726154 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:35.825590 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.825565 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:05:35.887368 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.887244 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.908926 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-kube-api-access-sztjc\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-kube-api-access-sztjc\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.908980 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-socket\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.909032 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-envoy\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.909078 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-istio-token\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-istio-token\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.909128 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-certs\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.909173 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e77602a7-cf53-413f-9484-7baccce25ea5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77602a7-cf53-413f-9484-7baccce25ea5-istio-podinfo\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.909311 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-data\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.909352 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77602a7-cf53-413f-9484-7baccce25ea5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77602a7-cf53-413f-9484-7baccce25ea5-istiod-ca-cert\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.909549 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.909414 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-credential-socket\") pod \"e77602a7-cf53-413f-9484-7baccce25ea5\" (UID: \"e77602a7-cf53-413f-9484-7baccce25ea5\") " Oct 29 23:05:35.910607 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.910473 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-socket" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:35.911147 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.910958 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-credential-socket" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:35.911343 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.911309 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-data" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:35.911520 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.911493 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-certs" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:35.911671 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.911648 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77602a7-cf53-413f-9484-7baccce25ea5-istiod-ca-cert" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:35.953249 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.912878 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:35.953249 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.912910 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:35.953249 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.912926 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:35.953249 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.912938 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77602a7-cf53-413f-9484-7baccce25ea5-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:35.953249 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.912951 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:35.953249 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.920298 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e77602a7-cf53-413f-9484-7baccce25ea5-istio-podinfo" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:05:35.955003 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.954941 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-envoy" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:05:35.961020 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.960984 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-kube-api-access-sztjc" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "kube-api-access-sztjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:35.961958 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:35.961931 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-istio-token" pod "e77602a7-cf53-413f-9484-7baccce25ea5" (UID: "e77602a7-cf53-413f-9484-7baccce25ea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:36.015209 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.014985 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sztjc\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-kube-api-access-sztjc\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:36.015209 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.015025 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77602a7-cf53-413f-9484-7baccce25ea5-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:36.015209 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.015052 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77602a7-cf53-413f-9484-7baccce25ea5-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:36.015209 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.015066 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77602a7-cf53-413f-9484-7baccce25ea5-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:36.075379 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.073742 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:36.576662 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.576622 5610 generic.go:356] "Generic (PLEG): container finished" podID="e77602a7-cf53-413f-9484-7baccce25ea5" containerID="a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def" exitCode=0 Oct 29 23:05:36.577163 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.576696 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" Oct 29 23:05:36.577163 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.576712 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" event={"ID":"e77602a7-cf53-413f-9484-7baccce25ea5","Type":"ContainerDied","Data":"a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def"} Oct 29 23:05:36.577163 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.576783 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw" event={"ID":"e77602a7-cf53-413f-9484-7baccce25ea5","Type":"ContainerDied","Data":"108cf3d74744bfe22280ae3fa7f94e40525123763fd0676c9e83a90d3d8f48f4"} Oct 29 23:05:36.577163 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.576814 5610 scope.go:122] "RemoveContainer" containerID="a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def" Oct 29 23:05:36.589644 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.589620 5610 scope.go:122] "RemoveContainer" containerID="d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa" Oct 29 23:05:36.601378 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.601350 5610 scope.go:122] "RemoveContainer" containerID="a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def" Oct 29 23:05:36.601724 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:36.601703 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def\": container with ID starting with a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def not found: ID does not exist" containerID="a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def" Oct 29 23:05:36.601795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.601735 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def"} err="failed to get container status \"a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def\": rpc error: code = NotFound desc = could not find container \"a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def\": container with ID starting with a72346854611269a0bcb539f76a32c552f56df199ce84d3c211271463e336def not found: ID does not exist" Oct 29 23:05:36.601795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.601754 5610 scope.go:122] "RemoveContainer" containerID="d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa" Oct 29 23:05:36.602028 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:36.602005 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa\": container with ID starting with d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa not found: ID does not exist" containerID="d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa" Oct 29 23:05:36.602082 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.602038 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa"} err="failed to get container status \"d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa\": rpc error: code = NotFound desc = could not find container \"d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa\": container with ID starting with d691c92d279d2d7301534d6df968971c64a3baddef786dbadce78d38dd95e7fa not found: ID does not exist" Oct 29 23:05:36.631301 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.631271 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw"] Oct 29 23:05:36.642619 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:36.642588 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ee1fc-openshift-default-587b4fdbd5-94skw"] Oct 29 23:05:37.304419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:37.302060 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:37.635564 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:37.635484 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" path="/var/lib/kubelet/pods/e77602a7-cf53-413f-9484-7baccce25ea5/volumes" Oct 29 23:05:38.695560 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:38.695475 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:40.333837 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:40.333780 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:41.201656 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:41.201625 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:41.852266 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:41.852210 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:42.003493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:42.003448 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" podUID="82fb649c-6c0b-456c-ae9b-c3e52195fc53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:05:42.014678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:42.014643 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" podUID="3a59f3f1-c569-4d63-9896-42d459c6f7bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:05:43.713379 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:43.713332 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:05:43.922965 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:43.922824 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:44.971203 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:44.971163 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254"] Oct 29 23:05:44.971722 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:44.971435 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" podUID="896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" containerName="maas-api" containerID="cri-o://f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8" gracePeriod=30 Oct 29 23:05:45.079651 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.079614 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:45.293206 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.293178 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 23:05:45.358430 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.357623 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-kube-api-access-bdkff\" (UniqueName: \"kubernetes.io/projected/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-kube-api-access-bdkff\") pod \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " Oct 29 23:05:45.358430 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.358162 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-metrics-certs\") pod \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " Oct 29 23:05:45.358703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.358523 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-maas-api-tls\") pod \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " Oct 29 23:05:45.358703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.358591 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-openshift-service-ca\") pod \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\" (UID: \"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a\") " Oct 29 23:05:45.361908 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.359546 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-openshift-service-ca" pod "896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" (UID: "896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:45.363552 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.362413 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-maas-api-tls" pod "896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" (UID: "896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:45.367410 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.364690 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-kube-api-access-bdkff" pod "896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" (UID: "896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a"). InnerVolumeSpecName "kube-api-access-bdkff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:45.372413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.371509 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-metrics-certs" pod "896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" (UID: "896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:45.460717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.460682 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdkff\" (UniqueName: \"kubernetes.io/projected/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-kube-api-access-bdkff\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:45.460717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.460712 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:45.460717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.460723 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:45.460975 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.460733 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:45.572855 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.572768 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:45.638938 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.638897 5610 generic.go:356] "Generic (PLEG): container finished" podID="896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" containerID="f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8" exitCode=0 Oct 29 23:05:45.639148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.638958 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" Oct 29 23:05:45.639148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.638985 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" event={"ID":"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a","Type":"ContainerDied","Data":"f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8"} Oct 29 23:05:45.639148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.639051 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254" event={"ID":"896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a","Type":"ContainerDied","Data":"2dc138fb4bd726fcdb985db0dce171ee7a61271e15ab33d96d584cb1bf1f40f1"} Oct 29 23:05:45.639148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.639083 5610 scope.go:122] "RemoveContainer" containerID="f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8" Oct 29 23:05:45.652674 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.652651 5610 scope.go:122] "RemoveContainer" containerID="f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8" Oct 29 23:05:45.653005 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:45.652982 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8\": container with ID starting with f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8 not found: ID does not exist" containerID="f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8" Oct 29 23:05:45.653099 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.653014 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8"} err="failed to get container status \"f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8\": rpc error: code = NotFound desc = could not find container \"f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8\": container with ID starting with f7d60851a4342e2e1f892a8b57d77e5814ab4488162eba82b18b8cbd5a1d5fe8 not found: ID does not exist" Oct 29 23:05:45.697720 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.697671 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254"] Oct 29 23:05:45.702875 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.701908 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-0d12c5-6d987bb458-v7254"] Oct 29 23:05:45.721135 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:45.721100 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:46.256283 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:46.256250 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:47.464220 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:47.464184 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:47.617947 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:47.617915 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:47.648978 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:47.648945 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" path="/var/lib/kubelet/pods/896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a/volumes" Oct 29 23:05:47.774743 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:47.774665 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:47.922313 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:47.922185 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:48.072291 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:48.072207 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:48.224769 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:48.224732 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:48.368600 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:48.368515 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:49.126801 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:49.126760 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:49.289675 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:49.289628 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:50.828727 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:50.828371 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:52.003856 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.003806 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" podUID="82fb649c-6c0b-456c-ae9b-c3e52195fc53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:05:52.014488 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.014457 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" podUID="3a59f3f1-c569-4d63-9896-42d459c6f7bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:05:52.243152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.243121 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:52.398584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.395370 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:52.512965 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.512938 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 23:05:52.546139 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.546112 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 23:05:52.627745 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.627660 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:52.654860 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.654827 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fb649c-6c0b-456c-ae9b-c3e52195fc53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fb649c-6c0b-456c-ae9b-c3e52195fc53-plugins-config-volume\") pod \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\" (UID: \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\") " Oct 29 23:05:52.655051 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.654888 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a59f3f1-c569-4d63-9896-42d459c6f7bd-kube-api-access-wpf6h\" (UniqueName: \"kubernetes.io/projected/3a59f3f1-c569-4d63-9896-42d459c6f7bd-kube-api-access-wpf6h\") pod \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\" (UID: \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\") " Oct 29 23:05:52.655051 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.654912 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fb649c-6c0b-456c-ae9b-c3e52195fc53-kube-api-access-rxv42\" (UniqueName: \"kubernetes.io/projected/82fb649c-6c0b-456c-ae9b-c3e52195fc53-kube-api-access-rxv42\") pod \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\" (UID: \"82fb649c-6c0b-456c-ae9b-c3e52195fc53\") " Oct 29 23:05:52.655051 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.654928 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a59f3f1-c569-4d63-9896-42d459c6f7bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a59f3f1-c569-4d63-9896-42d459c6f7bd-plugins-config-volume\") pod \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\" (UID: \"3a59f3f1-c569-4d63-9896-42d459c6f7bd\") " Oct 29 23:05:52.655360 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.655336 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fb649c-6c0b-456c-ae9b-c3e52195fc53-plugins-config-volume" pod "82fb649c-6c0b-456c-ae9b-c3e52195fc53" (UID: "82fb649c-6c0b-456c-ae9b-c3e52195fc53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:52.655488 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.655423 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a59f3f1-c569-4d63-9896-42d459c6f7bd-plugins-config-volume" pod "3a59f3f1-c569-4d63-9896-42d459c6f7bd" (UID: "3a59f3f1-c569-4d63-9896-42d459c6f7bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:52.655561 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.655547 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a59f3f1-c569-4d63-9896-42d459c6f7bd-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:52.655603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.655563 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fb649c-6c0b-456c-ae9b-c3e52195fc53-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:52.656992 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.656970 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a59f3f1-c569-4d63-9896-42d459c6f7bd-kube-api-access-wpf6h" pod "3a59f3f1-c569-4d63-9896-42d459c6f7bd" (UID: "3a59f3f1-c569-4d63-9896-42d459c6f7bd"). InnerVolumeSpecName "kube-api-access-wpf6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:52.657054 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.657038 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fb649c-6c0b-456c-ae9b-c3e52195fc53-kube-api-access-rxv42" pod "82fb649c-6c0b-456c-ae9b-c3e52195fc53" (UID: "82fb649c-6c0b-456c-ae9b-c3e52195fc53"). InnerVolumeSpecName "kube-api-access-rxv42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:52.687657 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.687623 5610 generic.go:356] "Generic (PLEG): container finished" podID="82fb649c-6c0b-456c-ae9b-c3e52195fc53" containerID="c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45" exitCode=0 Oct 29 23:05:52.687795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.687679 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" Oct 29 23:05:52.687795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.687710 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" event={"ID":"82fb649c-6c0b-456c-ae9b-c3e52195fc53","Type":"ContainerDied","Data":"c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45"} Oct 29 23:05:52.687795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.687766 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m" event={"ID":"82fb649c-6c0b-456c-ae9b-c3e52195fc53","Type":"ContainerDied","Data":"1ae4f1de37201a242f2534eaef17ffdfcac79792ab07eec1497dc478ddcc8bfb"} Oct 29 23:05:52.687994 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.687796 5610 scope.go:122] "RemoveContainer" containerID="c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45" Oct 29 23:05:52.705839 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.689725 5610 generic.go:356] "Generic (PLEG): container finished" podID="3a59f3f1-c569-4d63-9896-42d459c6f7bd" containerID="83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976" exitCode=0 Oct 29 23:05:52.705839 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.689775 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" event={"ID":"3a59f3f1-c569-4d63-9896-42d459c6f7bd","Type":"ContainerDied","Data":"83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976"} Oct 29 23:05:52.705839 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.689807 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" event={"ID":"3a59f3f1-c569-4d63-9896-42d459c6f7bd","Type":"ContainerDied","Data":"4d791f2fbb3ef559841f59a7efc6544d1a00b36b67c0cad35d8fd3bda3abe22f"} Oct 29 23:05:52.705839 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.689868 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6" Oct 29 23:05:52.764317 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.751582 5610 scope.go:122] "RemoveContainer" containerID="c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45" Oct 29 23:05:52.764317 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:52.752033 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45\": container with ID starting with c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45 not found: ID does not exist" containerID="c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45" Oct 29 23:05:52.764317 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.752073 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45"} err="failed to get container status \"c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45\": rpc error: code = NotFound desc = could not find container \"c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45\": container with ID starting with c2c74e85528533e47d0683e49008ccbfd95fcbd2ed7a2a732284d7325900ce45 not found: ID does not exist" Oct 29 23:05:52.764317 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.752096 5610 scope.go:122] "RemoveContainer" containerID="83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976" Oct 29 23:05:52.764317 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.756913 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxv42\" (UniqueName: \"kubernetes.io/projected/82fb649c-6c0b-456c-ae9b-c3e52195fc53-kube-api-access-rxv42\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:52.764317 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.756936 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpf6h\" (UniqueName: \"kubernetes.io/projected/3a59f3f1-c569-4d63-9896-42d459c6f7bd-kube-api-access-wpf6h\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:52.765860 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.765789 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6"] Oct 29 23:05:52.771673 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.771648 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-e81b56-78cd9675f9-6v6j6"] Oct 29 23:05:52.776852 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.776826 5610 scope.go:122] "RemoveContainer" containerID="83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976" Oct 29 23:05:52.777557 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:52.777538 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976\": container with ID starting with 83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976 not found: ID does not exist" containerID="83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976" Oct 29 23:05:52.777696 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.777677 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976"} err="failed to get container status \"83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976\": rpc error: code = NotFound desc = could not find container \"83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976\": container with ID starting with 83fcc496e8365856316010cfcb80ab5a2077b2fadd58c74173676f1d6515c976 not found: ID does not exist" Oct 29 23:05:52.806124 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.806096 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m"] Oct 29 23:05:52.810899 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:52.810842 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-e81b56-598655d846-svf7m"] Oct 29 23:05:53.635830 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.635797 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a59f3f1-c569-4d63-9896-42d459c6f7bd" path="/var/lib/kubelet/pods/3a59f3f1-c569-4d63-9896-42d459c6f7bd/volumes" Oct 29 23:05:53.636229 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.636215 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fb649c-6c0b-456c-ae9b-c3e52195fc53" path="/var/lib/kubelet/pods/82fb649c-6c0b-456c-ae9b-c3e52195fc53/volumes" Oct 29 23:05:53.643991 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.643965 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:53.669281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.669244 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677"] Oct 29 23:05:53.669552 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.669517 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" podUID="3b832c48-65fb-4aa1-b3fe-80e242f6e134" containerName="payload-processing" containerID="cri-o://0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045" gracePeriod=30 Oct 29 23:05:53.705344 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.705311 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx"] Oct 29 23:05:53.707844 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.707805 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" podUID="3331beb8-c6f0-4036-a335-93ca6648616f" containerName="payload-pre-processing" containerID="cri-o://1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632" gracePeriod=30 Oct 29 23:05:53.792578 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.792534 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:53.945588 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:53.945549 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:54.012745 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.012426 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:05:54.057687 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.057615 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:05:54.124556 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.124376 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:54.176292 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.176254 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3331beb8-c6f0-4036-a335-93ca6648616f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3331beb8-c6f0-4036-a335-93ca6648616f-plugins-config-volume\") pod \"3331beb8-c6f0-4036-a335-93ca6648616f\" (UID: \"3331beb8-c6f0-4036-a335-93ca6648616f\") " Oct 29 23:05:54.176513 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.176325 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b832c48-65fb-4aa1-b3fe-80e242f6e134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b832c48-65fb-4aa1-b3fe-80e242f6e134-plugins-config-volume\") pod \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\" (UID: \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\") " Oct 29 23:05:54.176513 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.176476 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b832c48-65fb-4aa1-b3fe-80e242f6e134-kube-api-access-7c7xv\" (UniqueName: \"kubernetes.io/projected/3b832c48-65fb-4aa1-b3fe-80e242f6e134-kube-api-access-7c7xv\") pod \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\" (UID: \"3b832c48-65fb-4aa1-b3fe-80e242f6e134\") " Oct 29 23:05:54.176678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.176657 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3331beb8-c6f0-4036-a335-93ca6648616f-kube-api-access-28t25\" (UniqueName: \"kubernetes.io/projected/3331beb8-c6f0-4036-a335-93ca6648616f-kube-api-access-28t25\") pod \"3331beb8-c6f0-4036-a335-93ca6648616f\" (UID: \"3331beb8-c6f0-4036-a335-93ca6648616f\") " Oct 29 23:05:54.176766 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.176739 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3331beb8-c6f0-4036-a335-93ca6648616f-plugins-config-volume" pod "3331beb8-c6f0-4036-a335-93ca6648616f" (UID: "3331beb8-c6f0-4036-a335-93ca6648616f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:54.177071 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.177052 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3331beb8-c6f0-4036-a335-93ca6648616f-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:54.177169 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.177091 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b832c48-65fb-4aa1-b3fe-80e242f6e134-plugins-config-volume" pod "3b832c48-65fb-4aa1-b3fe-80e242f6e134" (UID: "3b832c48-65fb-4aa1-b3fe-80e242f6e134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:54.179112 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.179087 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3331beb8-c6f0-4036-a335-93ca6648616f-kube-api-access-28t25" pod "3331beb8-c6f0-4036-a335-93ca6648616f" (UID: "3331beb8-c6f0-4036-a335-93ca6648616f"). InnerVolumeSpecName "kube-api-access-28t25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:54.179220 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.179195 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b832c48-65fb-4aa1-b3fe-80e242f6e134-kube-api-access-7c7xv" pod "3b832c48-65fb-4aa1-b3fe-80e242f6e134" (UID: "3b832c48-65fb-4aa1-b3fe-80e242f6e134"). InnerVolumeSpecName "kube-api-access-7c7xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:54.294228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.286708 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28t25\" (UniqueName: \"kubernetes.io/projected/3331beb8-c6f0-4036-a335-93ca6648616f-kube-api-access-28t25\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:54.294228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.286743 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b832c48-65fb-4aa1-b3fe-80e242f6e134-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:54.294228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.286757 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7xv\" (UniqueName: \"kubernetes.io/projected/3b832c48-65fb-4aa1-b3fe-80e242f6e134-kube-api-access-7c7xv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:54.296784 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.296747 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:54.716984 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.716942 5610 generic.go:356] "Generic (PLEG): container finished" podID="3b832c48-65fb-4aa1-b3fe-80e242f6e134" containerID="0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045" exitCode=0 Oct 29 23:05:54.716984 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.716981 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" Oct 29 23:05:54.717540 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.717021 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" event={"ID":"3b832c48-65fb-4aa1-b3fe-80e242f6e134","Type":"ContainerDied","Data":"0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045"} Oct 29 23:05:54.717540 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.717080 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677" event={"ID":"3b832c48-65fb-4aa1-b3fe-80e242f6e134","Type":"ContainerDied","Data":"72675c93f7c99e4731652083def44b4e889d881293ce916d3a8cb53de0aac949"} Oct 29 23:05:54.717540 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.717115 5610 scope.go:122] "RemoveContainer" containerID="0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045" Oct 29 23:05:54.723262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.719309 5610 generic.go:356] "Generic (PLEG): container finished" podID="3331beb8-c6f0-4036-a335-93ca6648616f" containerID="1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632" exitCode=0 Oct 29 23:05:54.723262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.719369 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" event={"ID":"3331beb8-c6f0-4036-a335-93ca6648616f","Type":"ContainerDied","Data":"1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632"} Oct 29 23:05:54.723262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.719416 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" Oct 29 23:05:54.723262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.719434 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx" event={"ID":"3331beb8-c6f0-4036-a335-93ca6648616f","Type":"ContainerDied","Data":"e9aee75bc21c22e6bcbed9aa2103051573b5e4ff08efb10e9e6af80de4d84449"} Oct 29 23:05:54.728577 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.728547 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:54.728879 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.728861 5610 scope.go:122] "RemoveContainer" containerID="0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045" Oct 29 23:05:54.729119 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:54.729096 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045\": container with ID starting with 0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045 not found: ID does not exist" containerID="0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045" Oct 29 23:05:54.729168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.729130 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045"} err="failed to get container status \"0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045\": rpc error: code = NotFound desc = could not find container \"0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045\": container with ID starting with 0b5de7032b7b5135f84ec8a65b04a25144989098c204f9f0f63ea9e81f8df045 not found: ID does not exist" Oct 29 23:05:54.729168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.729150 5610 scope.go:122] "RemoveContainer" containerID="1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632" Oct 29 23:05:54.738381 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.738363 5610 scope.go:122] "RemoveContainer" containerID="1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632" Oct 29 23:05:54.738685 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:54.738666 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632\": container with ID starting with 1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632 not found: ID does not exist" containerID="1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632" Oct 29 23:05:54.738744 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.738693 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632"} err="failed to get container status \"1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632\": rpc error: code = NotFound desc = could not find container \"1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632\": container with ID starting with 1bb7734d040d710028aded9adf6af7b3f99b23549f6e6cfbc0f8154f54045632 not found: ID does not exist" Oct 29 23:05:54.784245 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.784212 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677"] Oct 29 23:05:54.787335 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.787302 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eddb03-598b66b8d8-lc677"] Oct 29 23:05:54.822148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.822118 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx"] Oct 29 23:05:54.826461 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:54.826428 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eddb03-8576855547-rjdcx"] Oct 29 23:05:55.124646 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:55.124530 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:55.312345 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:55.312314 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:55.387369 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:55.387269 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6"] Oct 29 23:05:55.387595 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:55.387563 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" containerID="cri-o://74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308" gracePeriod=30 Oct 29 23:05:55.539776 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:55.539745 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:55.636199 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:55.636165 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3331beb8-c6f0-4036-a335-93ca6648616f" path="/var/lib/kubelet/pods/3331beb8-c6f0-4036-a335-93ca6648616f/volumes" Oct 29 23:05:55.636706 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:55.636683 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b832c48-65fb-4aa1-b3fe-80e242f6e134" path="/var/lib/kubelet/pods/3b832c48-65fb-4aa1-b3fe-80e242f6e134/volumes" Oct 29 23:05:56.083823 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:56.083789 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:56.833715 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:56.833672 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:57.811365 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:57.811328 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:58.883038 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:58.883001 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8"] Oct 29 23:05:58.883575 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:58.883256 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" podUID="2eee6ddd-7369-4268-b2fb-74c11ed01e3d" containerName="maas-api" containerID="cri-o://0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740" gracePeriod=30 Oct 29 23:05:58.981461 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:58.981423 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:59.023993 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.023953 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc"] Oct 29 23:05:59.024242 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.024211 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" podUID="65038ec5-df43-48fa-b380-d02bc2ec4a83" containerName="payload-processing" containerID="cri-o://c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009" gracePeriod=30 Oct 29 23:05:59.140174 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.140141 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:59.217997 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.217970 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:05:59.244126 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.244083 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-maas-api-tls\") pod \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " Oct 29 23:05:59.244308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.244165 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-metrics-certs\") pod \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " Oct 29 23:05:59.244308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.244266 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-openshift-service-ca\") pod \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " Oct 29 23:05:59.244308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.244295 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-kube-api-access-ql5nz\" (UniqueName: \"kubernetes.io/projected/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-kube-api-access-ql5nz\") pod \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\" (UID: \"2eee6ddd-7369-4268-b2fb-74c11ed01e3d\") " Oct 29 23:05:59.244871 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.244835 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-openshift-service-ca" pod "2eee6ddd-7369-4268-b2fb-74c11ed01e3d" (UID: "2eee6ddd-7369-4268-b2fb-74c11ed01e3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:59.247530 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.247502 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-metrics-certs" pod "2eee6ddd-7369-4268-b2fb-74c11ed01e3d" (UID: "2eee6ddd-7369-4268-b2fb-74c11ed01e3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:59.247655 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.247570 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-maas-api-tls" pod "2eee6ddd-7369-4268-b2fb-74c11ed01e3d" (UID: "2eee6ddd-7369-4268-b2fb-74c11ed01e3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:05:59.248234 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.248209 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-kube-api-access-ql5nz" pod "2eee6ddd-7369-4268-b2fb-74c11ed01e3d" (UID: "2eee6ddd-7369-4268-b2fb-74c11ed01e3d"). InnerVolumeSpecName "kube-api-access-ql5nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:59.287737 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.287602 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:59.334656 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.334626 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:59.345678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.345656 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65038ec5-df43-48fa-b380-d02bc2ec4a83-kube-api-access-lf8rv\" (UniqueName: \"kubernetes.io/projected/65038ec5-df43-48fa-b380-d02bc2ec4a83-kube-api-access-lf8rv\") pod \"65038ec5-df43-48fa-b380-d02bc2ec4a83\" (UID: \"65038ec5-df43-48fa-b380-d02bc2ec4a83\") " Oct 29 23:05:59.345759 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.345747 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65038ec5-df43-48fa-b380-d02bc2ec4a83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65038ec5-df43-48fa-b380-d02bc2ec4a83-plugins-config-volume\") pod \"65038ec5-df43-48fa-b380-d02bc2ec4a83\" (UID: \"65038ec5-df43-48fa-b380-d02bc2ec4a83\") " Oct 29 23:05:59.346025 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.346013 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.346074 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.346026 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql5nz\" (UniqueName: \"kubernetes.io/projected/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-kube-api-access-ql5nz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.346074 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.346035 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.346074 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.346044 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2eee6ddd-7369-4268-b2fb-74c11ed01e3d-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.346169 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.346095 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65038ec5-df43-48fa-b380-d02bc2ec4a83-plugins-config-volume" pod "65038ec5-df43-48fa-b380-d02bc2ec4a83" (UID: "65038ec5-df43-48fa-b380-d02bc2ec4a83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:05:59.347796 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.347776 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65038ec5-df43-48fa-b380-d02bc2ec4a83-kube-api-access-lf8rv" pod "65038ec5-df43-48fa-b380-d02bc2ec4a83" (UID: "65038ec5-df43-48fa-b380-d02bc2ec4a83"). InnerVolumeSpecName "kube-api-access-lf8rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:05:59.452473 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.447349 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf8rv\" (UniqueName: \"kubernetes.io/projected/65038ec5-df43-48fa-b380-d02bc2ec4a83-kube-api-access-lf8rv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.452473 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.447386 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65038ec5-df43-48fa-b380-d02bc2ec4a83-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:05:59.466850 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.466808 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:59.607972 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.607936 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:59.751641 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.751554 5610 generic.go:356] "Generic (PLEG): container finished" podID="65038ec5-df43-48fa-b380-d02bc2ec4a83" containerID="c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009" exitCode=0 Oct 29 23:05:59.751641 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.751602 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" Oct 29 23:05:59.751899 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.751594 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" event={"ID":"65038ec5-df43-48fa-b380-d02bc2ec4a83","Type":"ContainerDied","Data":"c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009"} Oct 29 23:05:59.751899 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.751728 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc" event={"ID":"65038ec5-df43-48fa-b380-d02bc2ec4a83","Type":"ContainerDied","Data":"75e1a162f10691c6877a39e01111c2e38f46d89fdd9d60725dc6d4f217ef4c7f"} Oct 29 23:05:59.751899 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.751763 5610 scope.go:122] "RemoveContainer" containerID="c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009" Oct 29 23:05:59.753054 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.753031 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:05:59.753266 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.753238 5610 generic.go:356] "Generic (PLEG): container finished" podID="2eee6ddd-7369-4268-b2fb-74c11ed01e3d" containerID="0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740" exitCode=0 Oct 29 23:05:59.753375 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.753295 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" Oct 29 23:05:59.753619 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.753463 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" event={"ID":"2eee6ddd-7369-4268-b2fb-74c11ed01e3d","Type":"ContainerDied","Data":"0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740"} Oct 29 23:05:59.753706 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.753686 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8" event={"ID":"2eee6ddd-7369-4268-b2fb-74c11ed01e3d","Type":"ContainerDied","Data":"3c005f2a4d9502f017154a40e51b3102f97b51030d739f94b18e35832bae7f06"} Oct 29 23:05:59.762515 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.762492 5610 scope.go:122] "RemoveContainer" containerID="c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009" Oct 29 23:05:59.762864 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:59.762840 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009\": container with ID starting with c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009 not found: ID does not exist" containerID="c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009" Oct 29 23:05:59.762948 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.762875 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009"} err="failed to get container status \"c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009\": rpc error: code = NotFound desc = could not find container \"c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009\": container with ID starting with c045e2bd521a2d8d7c514945e6878870364c07d8f9e918ca51a36dab53d58009 not found: ID does not exist" Oct 29 23:05:59.762948 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.762898 5610 scope.go:122] "RemoveContainer" containerID="0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740" Oct 29 23:05:59.772556 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.772537 5610 scope.go:122] "RemoveContainer" containerID="0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740" Oct 29 23:05:59.772974 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:05:59.772948 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740\": container with ID starting with 0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740 not found: ID does not exist" containerID="0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740" Oct 29 23:05:59.773109 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.773088 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740"} err="failed to get container status \"0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740\": rpc error: code = NotFound desc = could not find container \"0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740\": container with ID starting with 0e277958b09bc5490d997b6c4ba45228072f13b94db3764fbba8a4c6cc858740 not found: ID does not exist" Oct 29 23:05:59.805722 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.805689 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8"] Oct 29 23:05:59.810643 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.810614 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c74e55-76477947c7-7drm8"] Oct 29 23:05:59.844153 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.844112 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc"] Oct 29 23:05:59.850436 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:05:59.850385 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c74e55-59b8c6cc8-p2rgc"] Oct 29 23:06:00.095866 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.095779 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:00.696365 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.696330 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 23:06:00.765411 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.765358 5610 generic.go:356] "Generic (PLEG): container finished" podID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerID="74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308" exitCode=0 Oct 29 23:06:00.765620 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.765441 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" event={"ID":"9a39cdad-2f97-46b0-bf8b-9910a2963607","Type":"ContainerDied","Data":"74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308"} Oct 29 23:06:00.765620 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.765490 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" event={"ID":"9a39cdad-2f97-46b0-bf8b-9910a2963607","Type":"ContainerDied","Data":"e360a805a94e05ebed44673a5f12d00a767d012b1f77b4bb1ad034e710966700"} Oct 29 23:06:00.765620 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.765521 5610 scope.go:122] "RemoveContainer" containerID="74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308" Oct 29 23:06:00.765828 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.765635 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6" Oct 29 23:06:00.767231 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767097 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-podinfo\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.767231 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767148 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-credential-socket\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.767371 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767228 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-socket\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.767371 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767305 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-data\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.767371 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767359 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a39cdad-2f97-46b0-bf8b-9910a2963607-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a39cdad-2f97-46b0-bf8b-9910a2963607-istiod-ca-cert\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.767574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767445 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-token\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-token\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.767574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767519 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-certs\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.767574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767547 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-kube-api-access-js9th\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-kube-api-access-js9th\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.767749 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.767608 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-envoy\") pod \"9a39cdad-2f97-46b0-bf8b-9910a2963607\" (UID: \"9a39cdad-2f97-46b0-bf8b-9910a2963607\") " Oct 29 23:06:00.768496 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.768468 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-credential-socket" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.768624 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.768527 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-data" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.768876 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.768834 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-certs" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.769711 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.769684 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-socket" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.769873 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.769852 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a39cdad-2f97-46b0-bf8b-9910a2963607-istiod-ca-cert" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:00.771042 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.771007 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-podinfo" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:00.771890 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.771860 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-token" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:00.772124 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.772102 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-kube-api-access-js9th" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "kube-api-access-js9th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:00.772523 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.772491 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-envoy" pod "9a39cdad-2f97-46b0-bf8b-9910a2963607" (UID: "9a39cdad-2f97-46b0-bf8b-9910a2963607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:00.780510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.780086 5610 scope.go:122] "RemoveContainer" containerID="74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308" Oct 29 23:06:00.780578 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:00.780496 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308\": container with ID starting with 74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308 not found: ID does not exist" containerID="74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308" Oct 29 23:06:00.780578 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.780530 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308"} err="failed to get container status \"74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308\": rpc error: code = NotFound desc = could not find container \"74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308\": container with ID starting with 74e74bf67ed25977c217699fe43e30ce03f734031f1ee75ef8c4e74558b37308 not found: ID does not exist" Oct 29 23:06:00.869493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869447 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a39cdad-2f97-46b0-bf8b-9910a2963607-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.869493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869490 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.869493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869500 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.869493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869509 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js9th\" (UniqueName: \"kubernetes.io/projected/9a39cdad-2f97-46b0-bf8b-9910a2963607-kube-api-access-js9th\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.869493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869518 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.869493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869526 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.869999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869535 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.869999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869543 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:00.869999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:00.869553 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a39cdad-2f97-46b0-bf8b-9910a2963607-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:01.124204 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:01.124169 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6"] Oct 29 23:06:01.136805 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:01.136768 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-e81b56-openshift-default-58d77f7d69-c4wd6"] Oct 29 23:06:01.438984 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:01.438883 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:01.596604 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:01.596567 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:01.637799 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:01.637756 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eee6ddd-7369-4268-b2fb-74c11ed01e3d" path="/var/lib/kubelet/pods/2eee6ddd-7369-4268-b2fb-74c11ed01e3d/volumes" Oct 29 23:06:01.638421 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:01.638383 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65038ec5-df43-48fa-b380-d02bc2ec4a83" path="/var/lib/kubelet/pods/65038ec5-df43-48fa-b380-d02bc2ec4a83/volumes" Oct 29 23:06:01.639076 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:01.639056 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" path="/var/lib/kubelet/pods/9a39cdad-2f97-46b0-bf8b-9910a2963607/volumes" Oct 29 23:06:01.794958 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:01.794909 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:03.319811 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:03.319725 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:04.845099 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:04.845058 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:06.360765 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:06.360720 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:06.511330 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:06.511287 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs"] Oct 29 23:06:06.677123 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:06.677025 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:06.854576 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:06.854544 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:06.999123 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:06.999091 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:07.214628 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:07.214588 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:07.935290 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:07.933265 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:09.461195 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:09.461112 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:10.835557 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:10.835518 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:11.512347 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:11.512291 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" podUID="cdc1edc1-003a-4e57-9ab6-8c789418cb15" containerName="payload-pre-processing" containerID="cri-o://0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f" gracePeriod=25 Oct 29 23:06:12.215065 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:12.215002 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:12.676733 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:12.676639 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq"] Oct 29 23:06:12.713060 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:12.713027 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc"] Oct 29 23:06:12.878574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:12.878521 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:13.032990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:13.032946 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:13.271581 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:13.271540 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" podUID="cdc1edc1-003a-4e57-9ab6-8c789418cb15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:13.713897 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:13.713856 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:06:13.736471 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:13.736351 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:13.898619 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:13.898586 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:15.123883 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:15.123845 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:16.155492 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.155453 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv"] Oct 29 23:06:16.155999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.155720 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" podUID="1ff95e20-7adf-4a33-821d-1d51c58e0d58" containerName="maas-api" containerID="cri-o://759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75" gracePeriod=30 Oct 29 23:06:16.273540 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.273385 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:16.453485 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.453455 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 23:06:16.576562 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.576519 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-maas-api-tls\") pod \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " Oct 29 23:06:16.576768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.576599 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff95e20-7adf-4a33-821d-1d51c58e0d58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff95e20-7adf-4a33-821d-1d51c58e0d58-openshift-service-ca\") pod \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " Oct 29 23:06:16.576768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.576718 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff95e20-7adf-4a33-821d-1d51c58e0d58-kube-api-access-2tz2x\" (UniqueName: \"kubernetes.io/projected/1ff95e20-7adf-4a33-821d-1d51c58e0d58-kube-api-access-2tz2x\") pod \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " Oct 29 23:06:16.576768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.576749 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-metrics-certs\") pod \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\" (UID: \"1ff95e20-7adf-4a33-821d-1d51c58e0d58\") " Oct 29 23:06:16.578218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.577112 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff95e20-7adf-4a33-821d-1d51c58e0d58-openshift-service-ca" pod "1ff95e20-7adf-4a33-821d-1d51c58e0d58" (UID: "1ff95e20-7adf-4a33-821d-1d51c58e0d58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:16.578218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.577248 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff95e20-7adf-4a33-821d-1d51c58e0d58-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:16.580196 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.580169 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-metrics-certs" pod "1ff95e20-7adf-4a33-821d-1d51c58e0d58" (UID: "1ff95e20-7adf-4a33-821d-1d51c58e0d58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:16.580522 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.580495 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff95e20-7adf-4a33-821d-1d51c58e0d58-kube-api-access-2tz2x" pod "1ff95e20-7adf-4a33-821d-1d51c58e0d58" (UID: "1ff95e20-7adf-4a33-821d-1d51c58e0d58"). InnerVolumeSpecName "kube-api-access-2tz2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:16.580860 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.580839 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-maas-api-tls" pod "1ff95e20-7adf-4a33-821d-1d51c58e0d58" (UID: "1ff95e20-7adf-4a33-821d-1d51c58e0d58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:16.678300 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.678217 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tz2x\" (UniqueName: \"kubernetes.io/projected/1ff95e20-7adf-4a33-821d-1d51c58e0d58-kube-api-access-2tz2x\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:16.678556 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.678535 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:16.678667 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.678654 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ff95e20-7adf-4a33-821d-1d51c58e0d58-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:16.689754 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.689364 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:16.869763 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.869722 5610 generic.go:356] "Generic (PLEG): container finished" podID="1ff95e20-7adf-4a33-821d-1d51c58e0d58" containerID="759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75" exitCode=0 Oct 29 23:06:16.869973 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.869767 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" Oct 29 23:06:16.869973 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.869802 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" event={"ID":"1ff95e20-7adf-4a33-821d-1d51c58e0d58","Type":"ContainerDied","Data":"759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75"} Oct 29 23:06:16.869973 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.869858 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv" event={"ID":"1ff95e20-7adf-4a33-821d-1d51c58e0d58","Type":"ContainerDied","Data":"ce4f48145cfb232a520c3f59cc0d0b3af7bf7f048f5ad80b7ad519a1084842f7"} Oct 29 23:06:16.869973 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.869888 5610 scope.go:122] "RemoveContainer" containerID="759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75" Oct 29 23:06:16.880224 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.880204 5610 scope.go:122] "RemoveContainer" containerID="759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75" Oct 29 23:06:16.880541 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:16.880520 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75\": container with ID starting with 759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75 not found: ID does not exist" containerID="759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75" Oct 29 23:06:16.880663 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.880549 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75"} err="failed to get container status \"759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75\": rpc error: code = NotFound desc = could not find container \"759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75\": container with ID starting with 759cc72cbf2920a7f5948007122e1ed205614f62616c5b6ebacf740ea650ea75 not found: ID does not exist" Oct 29 23:06:16.882584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.882548 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:16.918711 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.918679 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv"] Oct 29 23:06:16.923720 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:16.923697 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-9a15f4-6c9b9f6879-5mqwv"] Oct 29 23:06:17.025192 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:17.025159 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:17.468599 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:17.468569 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:17.637098 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:17.637061 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff95e20-7adf-4a33-821d-1d51c58e0d58" path="/var/lib/kubelet/pods/1ff95e20-7adf-4a33-821d-1d51c58e0d58/volumes" Oct 29 23:06:17.677233 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:17.677187 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" podUID="dcfc3c40-8c05-4bce-885f-424f8742c844" containerName="payload-processing" containerID="cri-o://d73231b173d83d3699e1153109d31c4d0b54a1be036d6fc2f9e3203cd682cd20" gracePeriod=25 Oct 29 23:06:17.713339 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:17.713289 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" podUID="38e06ae3-0c7d-4158-9e15-63e75c718dc0" containerName="payload-pre-processing" containerID="cri-o://bb5b33f43de08c384241ad911e0074b8c93c0255fe0c28793bc714b731de794a" gracePeriod=25 Oct 29 23:06:18.330726 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:18.330689 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:18.475360 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:18.475313 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:18.752594 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:18.752560 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:18.902461 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:18.895989 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:19.359302 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:19.359203 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:20.380008 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:20.379971 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:20.582876 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:20.582841 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" podUID="dcfc3c40-8c05-4bce-885f-424f8742c844" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:20.583088 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:20.582895 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" podUID="38e06ae3-0c7d-4158-9e15-63e75c718dc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:21.796988 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:21.796657 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:23.189116 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:23.189078 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:23.271477 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:23.271436 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" podUID="cdc1edc1-003a-4e57-9ab6-8c789418cb15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:23.515706 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:23.515667 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:23.691496 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:23.691457 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:23.898607 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:23.898527 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln"] Oct 29 23:06:23.898770 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:23.898741 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" containerID="cri-o://507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad" gracePeriod=30 Oct 29 23:06:24.022435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:24.022389 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:25.029464 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:25.029419 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:25.879696 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:25.879648 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:26.026717 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:26.026682 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:26.257242 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:26.257205 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:26.686616 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:26.686522 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:26.834126 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:26.834087 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:28.003342 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.003306 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq"] Oct 29 23:06:28.003815 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.003531 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" podUID="60c6778f-8a94-4a33-bc9e-2ed413fd0d70" containerName="maas-api" containerID="cri-o://0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a" gracePeriod=30 Oct 29 23:06:28.107851 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.107808 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" podUID="60c6778f-8a94-4a33-bc9e-2ed413fd0d70" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.95:8443/health\": dial tcp 10.134.0.95:8443: connect: connection refused" Oct 29 23:06:28.195719 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.195656 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:28.285674 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.285649 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 23:06:28.344624 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.344589 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-kube-api-access-jdkst\" (UniqueName: \"kubernetes.io/projected/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-kube-api-access-jdkst\") pod \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " Oct 29 23:06:28.344820 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.344696 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-maas-api-tls\") pod \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " Oct 29 23:06:28.344820 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.344738 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-openshift-service-ca\") pod \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " Oct 29 23:06:28.345019 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.344938 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-metrics-certs\") pod \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\" (UID: \"60c6778f-8a94-4a33-bc9e-2ed413fd0d70\") " Oct 29 23:06:28.345167 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.345139 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-openshift-service-ca" pod "60c6778f-8a94-4a33-bc9e-2ed413fd0d70" (UID: "60c6778f-8a94-4a33-bc9e-2ed413fd0d70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:28.345407 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.345361 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:28.347021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.346995 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-maas-api-tls" pod "60c6778f-8a94-4a33-bc9e-2ed413fd0d70" (UID: "60c6778f-8a94-4a33-bc9e-2ed413fd0d70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:28.347134 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.347114 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-metrics-certs" pod "60c6778f-8a94-4a33-bc9e-2ed413fd0d70" (UID: "60c6778f-8a94-4a33-bc9e-2ed413fd0d70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:28.347470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.347453 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-kube-api-access-jdkst" pod "60c6778f-8a94-4a33-bc9e-2ed413fd0d70" (UID: "60c6778f-8a94-4a33-bc9e-2ed413fd0d70"). InnerVolumeSpecName "kube-api-access-jdkst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:28.446424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.446365 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:28.446424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.446423 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdkst\" (UniqueName: \"kubernetes.io/projected/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-kube-api-access-jdkst\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:28.446642 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.446434 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c6778f-8a94-4a33-bc9e-2ed413fd0d70-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:28.662929 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.662836 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:28.812276 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.812245 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:28.951629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.951600 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:28.960659 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.960616 5610 generic.go:356] "Generic (PLEG): container finished" podID="60c6778f-8a94-4a33-bc9e-2ed413fd0d70" containerID="0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a" exitCode=0 Oct 29 23:06:28.960844 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.960662 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" Oct 29 23:06:28.960844 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.960701 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" event={"ID":"60c6778f-8a94-4a33-bc9e-2ed413fd0d70","Type":"ContainerDied","Data":"0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a"} Oct 29 23:06:28.960844 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.960760 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq" event={"ID":"60c6778f-8a94-4a33-bc9e-2ed413fd0d70","Type":"ContainerDied","Data":"b7a1b0db746b2d2f0953de643000f4213eb847ed29fbfd538d7d0010623705d0"} Oct 29 23:06:28.960844 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:28.960784 5610 scope.go:122] "RemoveContainer" containerID="0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a" Oct 29 23:06:29.011779 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.011567 5610 scope.go:122] "RemoveContainer" containerID="0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a" Oct 29 23:06:29.012094 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:29.011899 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a\": container with ID starting with 0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a not found: ID does not exist" containerID="0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a" Oct 29 23:06:29.012094 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.011927 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a"} err="failed to get container status \"0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a\": rpc error: code = NotFound desc = could not find container \"0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a\": container with ID starting with 0f78ddec4d8a7b926c46dfe1641c5c0b4cfe5adf1c82e1d63711b46d0fc5aa0a not found: ID does not exist" Oct 29 23:06:29.022278 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.022246 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq"] Oct 29 23:06:29.033110 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.033081 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-36b46d-5b955474f5-t5rgq"] Oct 29 23:06:29.164900 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.164871 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 23:06:29.255538 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255499 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-data\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.255746 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255594 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-certs\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.255746 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255642 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-socket\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.255746 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255708 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-istio-token\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-istio-token\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.255746 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255741 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29da9dd4-983c-4343-9439-005194023ce1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29da9dd4-983c-4343-9439-005194023ce1-istio-podinfo\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.256000 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255784 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-kube-api-access-r77tw\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-kube-api-access-r77tw\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.256000 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255826 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-data" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.256000 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255835 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-certs" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.256000 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255873 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-envoy\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.256000 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255922 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-credential-socket\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.256000 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255951 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29da9dd4-983c-4343-9439-005194023ce1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29da9dd4-983c-4343-9439-005194023ce1-istiod-ca-cert\") pod \"29da9dd4-983c-4343-9439-005194023ce1\" (UID: \"29da9dd4-983c-4343-9439-005194023ce1\") " Oct 29 23:06:29.256000 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.255971 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-socket" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.256521 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.256487 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29da9dd4-983c-4343-9439-005194023ce1-istiod-ca-cert" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:29.256521 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.256558 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.256521 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.256579 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.256521 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.256593 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.256521 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.256605 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-credential-socket" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.258285 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.258257 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-envoy" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:29.258585 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.258561 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29da9dd4-983c-4343-9439-005194023ce1-istio-podinfo" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:29.258684 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.258564 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-istio-token" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:29.258754 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.258738 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-kube-api-access-r77tw" pod "29da9dd4-983c-4343-9439-005194023ce1" (UID: "29da9dd4-983c-4343-9439-005194023ce1"). InnerVolumeSpecName "kube-api-access-r77tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:29.357768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.357725 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.357768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.357766 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29da9dd4-983c-4343-9439-005194023ce1-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.357982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.357782 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77tw\" (UniqueName: \"kubernetes.io/projected/29da9dd4-983c-4343-9439-005194023ce1-kube-api-access-r77tw\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.357982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.357796 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.357982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.357811 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29da9dd4-983c-4343-9439-005194023ce1-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.357982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.357825 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29da9dd4-983c-4343-9439-005194023ce1-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:29.386693 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.386655 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:29.636060 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.635979 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c6778f-8a94-4a33-bc9e-2ed413fd0d70" path="/var/lib/kubelet/pods/60c6778f-8a94-4a33-bc9e-2ed413fd0d70/volumes" Oct 29 23:06:29.969763 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.969726 5610 generic.go:356] "Generic (PLEG): container finished" podID="29da9dd4-983c-4343-9439-005194023ce1" containerID="507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad" exitCode=0 Oct 29 23:06:29.969954 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.969778 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" event={"ID":"29da9dd4-983c-4343-9439-005194023ce1","Type":"ContainerDied","Data":"507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad"} Oct 29 23:06:29.969954 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.969786 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" Oct 29 23:06:29.969954 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.969819 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln" event={"ID":"29da9dd4-983c-4343-9439-005194023ce1","Type":"ContainerDied","Data":"7b214a085f9aabfcf28dc4855b67fa762024e1cb624f88caccd1a3ec76f32f21"} Oct 29 23:06:29.969954 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.969841 5610 scope.go:122] "RemoveContainer" containerID="507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad" Oct 29 23:06:29.992383 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:29.992346 5610 scope.go:122] "RemoveContainer" containerID="f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec" Oct 29 23:06:30.003494 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.003469 5610 scope.go:122] "RemoveContainer" containerID="507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad" Oct 29 23:06:30.003825 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:30.003793 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad\": container with ID starting with 507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad not found: ID does not exist" containerID="507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad" Oct 29 23:06:30.003945 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.003835 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad"} err="failed to get container status \"507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad\": rpc error: code = NotFound desc = could not find container \"507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad\": container with ID starting with 507eca053b08cbc447ef87b03355707dde2edadcc1b7f52833e8e3205f4a66ad not found: ID does not exist" Oct 29 23:06:30.003945 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.003854 5610 scope.go:122] "RemoveContainer" containerID="f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec" Oct 29 23:06:30.004117 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:30.004099 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec\": container with ID starting with f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec not found: ID does not exist" containerID="f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec" Oct 29 23:06:30.004160 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.004121 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec"} err="failed to get container status \"f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec\": rpc error: code = NotFound desc = could not find container \"f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec\": container with ID starting with f7d89c20fca6b8d23d78d25e47ca0f523a071500b22bac0ca6e8146c607d68ec not found: ID does not exist" Oct 29 23:06:30.041585 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.041546 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln"] Oct 29 23:06:30.052979 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.052948 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a15f4-openshift-default-6b75fcbdbc-n2cln"] Oct 29 23:06:30.582831 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.582790 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" podUID="38e06ae3-0c7d-4158-9e15-63e75c718dc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:30.583355 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.583231 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" podUID="dcfc3c40-8c05-4bce-885f-424f8742c844" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:30.600077 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:30.600020 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:31.609799 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.609768 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc"] Oct 29 23:06:31.636596 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.636567 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29da9dd4-983c-4343-9439-005194023ce1" path="/var/lib/kubelet/pods/29da9dd4-983c-4343-9439-005194023ce1/volumes" Oct 29 23:06:31.763019 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.762980 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:31.792685 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.792654 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 23:06:31.884979 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.884945 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc1edc1-003a-4e57-9ab6-8c789418cb15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc1edc1-003a-4e57-9ab6-8c789418cb15-plugins-config-volume\") pod \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\" (UID: \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\") " Oct 29 23:06:31.885212 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.885013 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc1edc1-003a-4e57-9ab6-8c789418cb15-kube-api-access-c4t2t\" (UniqueName: \"kubernetes.io/projected/cdc1edc1-003a-4e57-9ab6-8c789418cb15-kube-api-access-c4t2t\") pod \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\" (UID: \"cdc1edc1-003a-4e57-9ab6-8c789418cb15\") " Oct 29 23:06:31.885429 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.885405 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc1edc1-003a-4e57-9ab6-8c789418cb15-plugins-config-volume" pod "cdc1edc1-003a-4e57-9ab6-8c789418cb15" (UID: "cdc1edc1-003a-4e57-9ab6-8c789418cb15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:31.887236 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.887216 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc1edc1-003a-4e57-9ab6-8c789418cb15-kube-api-access-c4t2t" pod "cdc1edc1-003a-4e57-9ab6-8c789418cb15" (UID: "cdc1edc1-003a-4e57-9ab6-8c789418cb15"). InnerVolumeSpecName "kube-api-access-c4t2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:31.984325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.984286 5610 generic.go:356] "Generic (PLEG): container finished" podID="cdc1edc1-003a-4e57-9ab6-8c789418cb15" containerID="0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f" exitCode=0 Oct 29 23:06:31.984551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.984335 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" Oct 29 23:06:31.984551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.984363 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" event={"ID":"cdc1edc1-003a-4e57-9ab6-8c789418cb15","Type":"ContainerDied","Data":"0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f"} Oct 29 23:06:31.984551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.984440 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs" event={"ID":"cdc1edc1-003a-4e57-9ab6-8c789418cb15","Type":"ContainerDied","Data":"4192836be480f5649b0bd14d18803517d4be9f56c51c47ab8b427ae8e4495b61"} Oct 29 23:06:31.984551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.984464 5610 scope.go:122] "RemoveContainer" containerID="0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f" Oct 29 23:06:31.986408 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.986370 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc1edc1-003a-4e57-9ab6-8c789418cb15-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.986408 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.986407 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4t2t\" (UniqueName: \"kubernetes.io/projected/cdc1edc1-003a-4e57-9ab6-8c789418cb15-kube-api-access-c4t2t\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:31.995212 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.995194 5610 scope.go:122] "RemoveContainer" containerID="0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f" Oct 29 23:06:31.995484 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:31.995465 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f\": container with ID starting with 0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f not found: ID does not exist" containerID="0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f" Oct 29 23:06:31.995543 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:31.995491 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f"} err="failed to get container status \"0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f\": rpc error: code = NotFound desc = could not find container \"0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f\": container with ID starting with 0c3c05aca6a999431081ca92e1ca90824418bef5b5abaded6d764f92971e1f9f not found: ID does not exist" Oct 29 23:06:32.034728 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:32.034630 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs"] Oct 29 23:06:32.038324 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:32.038295 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-6bb06a-69bfbb5cf-bbvcs"] Oct 29 23:06:32.117372 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:32.117236 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:33.636002 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:33.635963 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc1edc1-003a-4e57-9ab6-8c789418cb15" path="/var/lib/kubelet/pods/cdc1edc1-003a-4e57-9ab6-8c789418cb15/volumes" Oct 29 23:06:33.747132 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:33.747096 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:34.836254 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:34.836172 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn"] Oct 29 23:06:34.853311 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:34.853270 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd"] Oct 29 23:06:35.017209 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:35.017172 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:35.163585 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:35.163494 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:35.306409 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:35.306362 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:35.465427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:35.464296 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:35.615950 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:35.615913 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5"] Oct 29 23:06:35.616237 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:35.616208 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" containerID="cri-o://0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c" gracePeriod=30 Oct 29 23:06:35.699562 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:35.699528 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:36.775636 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:36.775607 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:37.423767 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:37.423735 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:37.572998 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:37.572965 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:38.023792 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.023682 5610 generic.go:356] "Generic (PLEG): container finished" podID="38e06ae3-0c7d-4158-9e15-63e75c718dc0" containerID="bb5b33f43de08c384241ad911e0074b8c93c0255fe0c28793bc714b731de794a" exitCode=0 Oct 29 23:06:38.023792 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.023752 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" event={"ID":"38e06ae3-0c7d-4158-9e15-63e75c718dc0","Type":"ContainerDied","Data":"bb5b33f43de08c384241ad911e0074b8c93c0255fe0c28793bc714b731de794a"} Oct 29 23:06:38.025465 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.025437 5610 generic.go:356] "Generic (PLEG): container finished" podID="dcfc3c40-8c05-4bce-885f-424f8742c844" containerID="d73231b173d83d3699e1153109d31c4d0b54a1be036d6fc2f9e3203cd682cd20" exitCode=0 Oct 29 23:06:38.025601 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.025490 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" event={"ID":"dcfc3c40-8c05-4bce-885f-424f8742c844","Type":"ContainerDied","Data":"d73231b173d83d3699e1153109d31c4d0b54a1be036d6fc2f9e3203cd682cd20"} Oct 29 23:06:38.025601 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.025534 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" event={"ID":"dcfc3c40-8c05-4bce-885f-424f8742c844","Type":"ContainerDied","Data":"3f9b3fd21a852873550cb2213e7d66eb30f2151fb7bfb2639b0752f4c62e5428"} Oct 29 23:06:38.025601 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.025551 5610 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f9b3fd21a852873550cb2213e7d66eb30f2151fb7bfb2639b0752f4c62e5428" Oct 29 23:06:38.033144 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.033125 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 23:06:38.059777 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.059725 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 23:06:38.150918 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.150870 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:38.170424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.170363 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e06ae3-0c7d-4158-9e15-63e75c718dc0-kube-api-access-mr5ns\" (UniqueName: \"kubernetes.io/projected/38e06ae3-0c7d-4158-9e15-63e75c718dc0-kube-api-access-mr5ns\") pod \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\" (UID: \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\") " Oct 29 23:06:38.170424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.170410 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfc3c40-8c05-4bce-885f-424f8742c844-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfc3c40-8c05-4bce-885f-424f8742c844-plugins-config-volume\") pod \"dcfc3c40-8c05-4bce-885f-424f8742c844\" (UID: \"dcfc3c40-8c05-4bce-885f-424f8742c844\") " Oct 29 23:06:38.170669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.170510 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e06ae3-0c7d-4158-9e15-63e75c718dc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e06ae3-0c7d-4158-9e15-63e75c718dc0-plugins-config-volume\") pod \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\" (UID: \"38e06ae3-0c7d-4158-9e15-63e75c718dc0\") " Oct 29 23:06:38.170669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.170609 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfc3c40-8c05-4bce-885f-424f8742c844-kube-api-access-qd8b6\" (UniqueName: \"kubernetes.io/projected/dcfc3c40-8c05-4bce-885f-424f8742c844-kube-api-access-qd8b6\") pod \"dcfc3c40-8c05-4bce-885f-424f8742c844\" (UID: \"dcfc3c40-8c05-4bce-885f-424f8742c844\") " Oct 29 23:06:38.170897 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.170870 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e06ae3-0c7d-4158-9e15-63e75c718dc0-plugins-config-volume" pod "38e06ae3-0c7d-4158-9e15-63e75c718dc0" (UID: "38e06ae3-0c7d-4158-9e15-63e75c718dc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:38.170988 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.170873 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfc3c40-8c05-4bce-885f-424f8742c844-plugins-config-volume" pod "dcfc3c40-8c05-4bce-885f-424f8742c844" (UID: "dcfc3c40-8c05-4bce-885f-424f8742c844"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:38.171111 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.171086 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38e06ae3-0c7d-4158-9e15-63e75c718dc0-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.171261 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.171111 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcfc3c40-8c05-4bce-885f-424f8742c844-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.172786 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.172760 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfc3c40-8c05-4bce-885f-424f8742c844-kube-api-access-qd8b6" pod "dcfc3c40-8c05-4bce-885f-424f8742c844" (UID: "dcfc3c40-8c05-4bce-885f-424f8742c844"). InnerVolumeSpecName "kube-api-access-qd8b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:38.172873 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.172842 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e06ae3-0c7d-4158-9e15-63e75c718dc0-kube-api-access-mr5ns" pod "38e06ae3-0c7d-4158-9e15-63e75c718dc0" (UID: "38e06ae3-0c7d-4158-9e15-63e75c718dc0"). InnerVolumeSpecName "kube-api-access-mr5ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:38.272592 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.272506 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr5ns\" (UniqueName: \"kubernetes.io/projected/38e06ae3-0c7d-4158-9e15-63e75c718dc0-kube-api-access-mr5ns\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.272592 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.272536 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd8b6\" (UniqueName: \"kubernetes.io/projected/dcfc3c40-8c05-4bce-885f-424f8742c844-kube-api-access-qd8b6\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:38.625699 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.625609 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:38.885340 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:38.885244 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:39.034223 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.034193 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" Oct 29 23:06:39.034755 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.034233 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc" event={"ID":"38e06ae3-0c7d-4158-9e15-63e75c718dc0","Type":"ContainerDied","Data":"0c842f06c6336e4d4f18752e70284ed7d5d9ba79825b8a6a82ba10d719bdaeef"} Oct 29 23:06:39.034755 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.034309 5610 scope.go:122] "RemoveContainer" containerID="bb5b33f43de08c384241ad911e0074b8c93c0255fe0c28793bc714b731de794a" Oct 29 23:06:39.034755 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.034571 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq" Oct 29 23:06:39.043745 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.043645 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:39.089068 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.089038 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq"] Oct 29 23:06:39.091599 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.091575 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9a15f4-779b664b68-sw7zq"] Oct 29 23:06:39.117743 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.117708 5610 patch_prober.go:29] interesting pod/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:39.117907 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.117761 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:39.132729 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.132701 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc"] Oct 29 23:06:39.137661 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.137603 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9a15f4-7c7d4bdbc5-mfbxc"] Oct 29 23:06:39.365661 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.365628 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:39.643367 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.643328 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e06ae3-0c7d-4158-9e15-63e75c718dc0" path="/var/lib/kubelet/pods/38e06ae3-0c7d-4158-9e15-63e75c718dc0/volumes" Oct 29 23:06:39.643786 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.643768 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfc3c40-8c05-4bce-885f-424f8742c844" path="/var/lib/kubelet/pods/dcfc3c40-8c05-4bce-885f-424f8742c844/volumes" Oct 29 23:06:39.837674 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.837612 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" podUID="12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" containerName="payload-processing" containerID="cri-o://5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab" gracePeriod=25 Oct 29 23:06:39.854212 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:39.854170 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" podUID="7cb82a55-445c-40e7-95b3-a40ba0688d71" containerName="payload-pre-processing" containerID="cri-o://926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee" gracePeriod=25 Oct 29 23:06:40.419266 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:40.419221 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:40.883220 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:40.883192 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 23:06:41.005830 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.005743 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-certs\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.005830 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.005798 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-kube-api-access-jn2jp\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-kube-api-access-jn2jp\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.006063 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.005861 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-istio-token\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-istio-token\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.006063 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.005890 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-socket\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.006063 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.005928 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-data\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.006063 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.005963 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-credential-socket\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.006063 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.005997 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55441a0e-ab81-44df-8301-956c67fecc3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55441a0e-ab81-44df-8301-956c67fecc3b-istio-podinfo\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.006063 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006038 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-envoy\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.006425 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006116 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-certs" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.006425 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006124 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55441a0e-ab81-44df-8301-956c67fecc3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55441a0e-ab81-44df-8301-956c67fecc3b-istiod-ca-cert\") pod \"55441a0e-ab81-44df-8301-956c67fecc3b\" (UID: \"55441a0e-ab81-44df-8301-956c67fecc3b\") " Oct 29 23:06:41.006425 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006337 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-credential-socket" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.006425 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006356 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-data" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.006684 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006655 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-socket" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.006752 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006731 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55441a0e-ab81-44df-8301-956c67fecc3b-istiod-ca-cert" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:41.006821 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006746 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.006821 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006772 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.006821 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006787 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.006821 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.006802 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.008496 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.008465 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-envoy" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:41.008709 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.008682 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55441a0e-ab81-44df-8301-956c67fecc3b-istio-podinfo" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:41.008855 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.008726 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-kube-api-access-jn2jp" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "kube-api-access-jn2jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:41.008855 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.008766 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-istio-token" pod "55441a0e-ab81-44df-8301-956c67fecc3b" (UID: "55441a0e-ab81-44df-8301-956c67fecc3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:41.050159 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.050127 5610 generic.go:356] "Generic (PLEG): container finished" podID="55441a0e-ab81-44df-8301-956c67fecc3b" containerID="0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c" exitCode=0 Oct 29 23:06:41.050316 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.050183 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" Oct 29 23:06:41.050316 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.050216 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" event={"ID":"55441a0e-ab81-44df-8301-956c67fecc3b","Type":"ContainerDied","Data":"0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c"} Oct 29 23:06:41.050316 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.050274 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5" event={"ID":"55441a0e-ab81-44df-8301-956c67fecc3b","Type":"ContainerDied","Data":"6a6351277ce043c7d38f702c408b2e62dc2fd12287dbed3003c95685b424c06f"} Oct 29 23:06:41.050316 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.050307 5610 scope.go:122] "RemoveContainer" containerID="0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c" Oct 29 23:06:41.060642 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.060617 5610 scope.go:122] "RemoveContainer" containerID="1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57" Oct 29 23:06:41.071415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.071382 5610 scope.go:122] "RemoveContainer" containerID="0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c" Oct 29 23:06:41.071757 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:41.071734 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c\": container with ID starting with 0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c not found: ID does not exist" containerID="0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c" Oct 29 23:06:41.071854 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.071765 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c"} err="failed to get container status \"0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c\": rpc error: code = NotFound desc = could not find container \"0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c\": container with ID starting with 0985df146d1691186cef2e439191dff37b3bf97b52a0b0dce9bed01c0136069c not found: ID does not exist" Oct 29 23:06:41.071854 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.071783 5610 scope.go:122] "RemoveContainer" containerID="1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57" Oct 29 23:06:41.072078 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:41.072061 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57\": container with ID starting with 1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57 not found: ID does not exist" containerID="1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57" Oct 29 23:06:41.072134 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.072082 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57"} err="failed to get container status \"1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57\": rpc error: code = NotFound desc = could not find container \"1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57\": container with ID starting with 1ff7b897e0e90696e5a1714e0010f99d19d165cf5d3cc8cbbc865ebc0658aa57 not found: ID does not exist" Oct 29 23:06:41.100148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.100116 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5"] Oct 29 23:06:41.103752 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.103723 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-36b46d-openshift-default-d5d76d987-9kqf5"] Oct 29 23:06:41.108384 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.108358 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55441a0e-ab81-44df-8301-956c67fecc3b-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.108384 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.108384 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55441a0e-ab81-44df-8301-956c67fecc3b-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.108583 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.108409 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55441a0e-ab81-44df-8301-956c67fecc3b-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.108583 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.108418 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2jp\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-kube-api-access-jn2jp\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.108583 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.108428 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55441a0e-ab81-44df-8301-956c67fecc3b-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:41.313037 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.312946 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:41.635539 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.635450 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" path="/var/lib/kubelet/pods/55441a0e-ab81-44df-8301-956c67fecc3b/volumes" Oct 29 23:06:41.941830 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:41.941710 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:43.307083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:43.307053 5610 scope.go:122] "RemoveContainer" containerID="93f3311c5ca5779afee2dfb5c7693854f5aec0e0ebc3e38d5ea6c7bb58ab20da" Oct 29 23:06:43.316724 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:43.316702 5610 scope.go:122] "RemoveContainer" containerID="d73231b173d83d3699e1153109d31c4d0b54a1be036d6fc2f9e3203cd682cd20" Oct 29 23:06:43.439825 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:43.439793 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:43.710576 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:43.710537 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:06:44.843632 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:44.843569 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:45.166218 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:45.166182 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:45.322578 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:45.322541 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:45.432152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:45.432066 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9"] Oct 29 23:06:45.432424 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:45.432325 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" containerID="cri-o://f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b" gracePeriod=30 Oct 29 23:06:45.531329 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:45.531283 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:46.062120 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.062083 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:46.533536 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.533468 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:46.617318 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.617272 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" podUID="12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:46.632487 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.632433 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" containerID="cri-o://4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7" gracePeriod=45 Oct 29 23:06:46.768702 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.768660 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:46.912449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.912418 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 23:06:46.939200 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.939163 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:46.991108 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991075 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-tmp-dir\") pod \"2d8052bf-194c-495c-a1d4-4431b30768ce\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " Oct 29 23:06:46.991317 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991127 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-home\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-home\") pod \"2d8052bf-194c-495c-a1d4-4431b30768ce\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " Oct 29 23:06:46.991317 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991220 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-dshm\") pod \"2d8052bf-194c-495c-a1d4-4431b30768ce\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " Oct 29 23:06:46.991317 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991277 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8052bf-194c-495c-a1d4-4431b30768ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8052bf-194c-495c-a1d4-4431b30768ce-tls-certs\") pod \"2d8052bf-194c-495c-a1d4-4431b30768ce\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " Oct 29 23:06:46.991520 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991356 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8052bf-194c-495c-a1d4-4431b30768ce-kube-api-access-dd6gq\" (UniqueName: \"kubernetes.io/projected/2d8052bf-194c-495c-a1d4-4431b30768ce-kube-api-access-dd6gq\") pod \"2d8052bf-194c-495c-a1d4-4431b30768ce\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " Oct 29 23:06:46.991520 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991453 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-model-cache\") pod \"2d8052bf-194c-495c-a1d4-4431b30768ce\" (UID: \"2d8052bf-194c-495c-a1d4-4431b30768ce\") " Oct 29 23:06:46.991520 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991456 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-tmp-dir" pod "2d8052bf-194c-495c-a1d4-4431b30768ce" (UID: "2d8052bf-194c-495c-a1d4-4431b30768ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:46.991795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991760 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-home" pod "2d8052bf-194c-495c-a1d4-4431b30768ce" (UID: "2d8052bf-194c-495c-a1d4-4431b30768ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:46.991942 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.991841 5610 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:46.992099 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.992067 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-model-cache" pod "2d8052bf-194c-495c-a1d4-4431b30768ce" (UID: "2d8052bf-194c-495c-a1d4-4431b30768ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:46.994342 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.994301 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-dshm" pod "2d8052bf-194c-495c-a1d4-4431b30768ce" (UID: "2d8052bf-194c-495c-a1d4-4431b30768ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:46.994647 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.994623 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8052bf-194c-495c-a1d4-4431b30768ce-tls-certs" pod "2d8052bf-194c-495c-a1d4-4431b30768ce" (UID: "2d8052bf-194c-495c-a1d4-4431b30768ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:46.994736 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:46.994655 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8052bf-194c-495c-a1d4-4431b30768ce-kube-api-access-dd6gq" pod "2d8052bf-194c-495c-a1d4-4431b30768ce" (UID: "2d8052bf-194c-495c-a1d4-4431b30768ce"). InnerVolumeSpecName "kube-api-access-dd6gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:47.092990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.092898 5610 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.092990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.092938 5610 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d8052bf-194c-495c-a1d4-4431b30768ce-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.092990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.092953 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6gq\" (UniqueName: \"kubernetes.io/projected/2d8052bf-194c-495c-a1d4-4431b30768ce-kube-api-access-dd6gq\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.092990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.092967 5610 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.092990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.092980 5610 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d8052bf-194c-495c-a1d4-4431b30768ce-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:47.095995 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.095958 5610 generic.go:356] "Generic (PLEG): container finished" podID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerID="4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7" exitCode=0 Oct 29 23:06:47.096148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.096000 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" Oct 29 23:06:47.096148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.096044 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" event={"ID":"2d8052bf-194c-495c-a1d4-4431b30768ce","Type":"ContainerDied","Data":"4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7"} Oct 29 23:06:47.096148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.096096 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc" event={"ID":"2d8052bf-194c-495c-a1d4-4431b30768ce","Type":"ContainerDied","Data":"09433dd988b324921d7ef3c864667f68016651ce6be788f727b3d3567f95edb0"} Oct 29 23:06:47.096148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.096118 5610 scope.go:122] "RemoveContainer" containerID="4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7" Oct 29 23:06:47.109110 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.109080 5610 scope.go:122] "RemoveContainer" containerID="4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7" Oct 29 23:06:47.109532 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:47.109505 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7\": container with ID starting with 4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7 not found: ID does not exist" containerID="4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7" Oct 29 23:06:47.109636 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.109539 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7"} err="failed to get container status \"4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7\": rpc error: code = NotFound desc = could not find container \"4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7\": container with ID starting with 4d9ed0ff30205675e7046652de2da15e07ec97163c1a9d789f02ea0a11323bc7 not found: ID does not exist" Oct 29 23:06:47.148715 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.148683 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc"] Oct 29 23:06:47.153201 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.153172 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-92fd86/distinct-w6-92fd86-kserve-9f776dd-gwphc"] Oct 29 23:06:47.628788 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.628749 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" podUID="7cb82a55-445c-40e7-95b3-a40ba0688d71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:47.635385 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.635360 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" path="/var/lib/kubelet/pods/2d8052bf-194c-495c-a1d4-4431b30768ce/volumes" Oct 29 23:06:47.965194 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.965128 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs"] Oct 29 23:06:47.965473 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:47.965372 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" podUID="f7d38d15-b38f-423b-aeef-68f36963738c" containerName="maas-api" containerID="cri-o://252cf9424398b9042c5340177e4929825ddb62cf40d105f547fdd3e2d31b6abf" gracePeriod=30 Oct 29 23:06:48.084731 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.084699 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:48.105567 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.105534 5610 generic.go:356] "Generic (PLEG): container finished" podID="f7d38d15-b38f-423b-aeef-68f36963738c" containerID="252cf9424398b9042c5340177e4929825ddb62cf40d105f547fdd3e2d31b6abf" exitCode=0 Oct 29 23:06:48.105929 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.105591 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" event={"ID":"f7d38d15-b38f-423b-aeef-68f36963738c","Type":"ContainerDied","Data":"252cf9424398b9042c5340177e4929825ddb62cf40d105f547fdd3e2d31b6abf"} Oct 29 23:06:48.251771 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.251741 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 23:06:48.305605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.304308 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d38d15-b38f-423b-aeef-68f36963738c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d38d15-b38f-423b-aeef-68f36963738c-openshift-service-ca\") pod \"f7d38d15-b38f-423b-aeef-68f36963738c\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " Oct 29 23:06:48.305605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.304431 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-maas-api-tls\") pod \"f7d38d15-b38f-423b-aeef-68f36963738c\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " Oct 29 23:06:48.305605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.304454 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-metrics-certs\") pod \"f7d38d15-b38f-423b-aeef-68f36963738c\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " Oct 29 23:06:48.305605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.304509 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d38d15-b38f-423b-aeef-68f36963738c-kube-api-access-8sbt2\" (UniqueName: \"kubernetes.io/projected/f7d38d15-b38f-423b-aeef-68f36963738c-kube-api-access-8sbt2\") pod \"f7d38d15-b38f-423b-aeef-68f36963738c\" (UID: \"f7d38d15-b38f-423b-aeef-68f36963738c\") " Oct 29 23:06:48.317414 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.307826 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d38d15-b38f-423b-aeef-68f36963738c-openshift-service-ca" pod "f7d38d15-b38f-423b-aeef-68f36963738c" (UID: "f7d38d15-b38f-423b-aeef-68f36963738c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:48.317414 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.309554 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d38d15-b38f-423b-aeef-68f36963738c-kube-api-access-8sbt2" pod "f7d38d15-b38f-423b-aeef-68f36963738c" (UID: "f7d38d15-b38f-423b-aeef-68f36963738c"). InnerVolumeSpecName "kube-api-access-8sbt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:48.317414 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.309597 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-maas-api-tls" pod "f7d38d15-b38f-423b-aeef-68f36963738c" (UID: "f7d38d15-b38f-423b-aeef-68f36963738c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:48.317414 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.312545 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-metrics-certs" pod "f7d38d15-b38f-423b-aeef-68f36963738c" (UID: "f7d38d15-b38f-423b-aeef-68f36963738c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:48.418228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.418183 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d38d15-b38f-423b-aeef-68f36963738c-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:48.418228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.418226 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:48.418228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.418240 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7d38d15-b38f-423b-aeef-68f36963738c-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:48.418228 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.418253 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sbt2\" (UniqueName: \"kubernetes.io/projected/f7d38d15-b38f-423b-aeef-68f36963738c-kube-api-access-8sbt2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:48.635257 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.635158 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:48.790950 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.790915 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:48.835705 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.835669 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:48.835897 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:48.835736 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:49.121248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.121209 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" Oct 29 23:06:49.121248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.121222 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs" event={"ID":"f7d38d15-b38f-423b-aeef-68f36963738c","Type":"ContainerDied","Data":"343fb2cba13a9433a3b57e74d1f5f9e3ac9e1d5b7ff5cd8e89ce2da635b9044f"} Oct 29 23:06:49.121803 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.121310 5610 scope.go:122] "RemoveContainer" containerID="252cf9424398b9042c5340177e4929825ddb62cf40d105f547fdd3e2d31b6abf" Oct 29 23:06:49.174479 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.174444 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs"] Oct 29 23:06:49.185306 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.185271 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8f1627-5d65447d9c-8rqfs"] Oct 29 23:06:49.292444 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.292409 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:49.636637 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.636593 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d38d15-b38f-423b-aeef-68f36963738c" path="/var/lib/kubelet/pods/f7d38d15-b38f-423b-aeef-68f36963738c/volumes" Oct 29 23:06:49.995057 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.995021 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts"] Oct 29 23:06:49.995302 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:49.995271 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" podUID="dc11cd91-f5b5-45d6-8504-2d56367411a9" containerName="maas-api" containerID="cri-o://7fe6e5fc0a1bb2b6d2e5750bf7ebdd52835e5591917c6b74a0dcf2d3205871fc" gracePeriod=30 Oct 29 23:06:50.100099 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.100061 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:50.130325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.130277 5610 generic.go:356] "Generic (PLEG): container finished" podID="dc11cd91-f5b5-45d6-8504-2d56367411a9" containerID="7fe6e5fc0a1bb2b6d2e5750bf7ebdd52835e5591917c6b74a0dcf2d3205871fc" exitCode=0 Oct 29 23:06:50.130841 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.130332 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" event={"ID":"dc11cd91-f5b5-45d6-8504-2d56367411a9","Type":"ContainerDied","Data":"7fe6e5fc0a1bb2b6d2e5750bf7ebdd52835e5591917c6b74a0dcf2d3205871fc"} Oct 29 23:06:50.285679 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.285641 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 23:06:50.345589 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.345555 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-maas-api-tls\") pod \"dc11cd91-f5b5-45d6-8504-2d56367411a9\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " Oct 29 23:06:50.345909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.345633 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc11cd91-f5b5-45d6-8504-2d56367411a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11cd91-f5b5-45d6-8504-2d56367411a9-openshift-service-ca\") pod \"dc11cd91-f5b5-45d6-8504-2d56367411a9\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " Oct 29 23:06:50.347309 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.347275 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc11cd91-f5b5-45d6-8504-2d56367411a9-openshift-service-ca" pod "dc11cd91-f5b5-45d6-8504-2d56367411a9" (UID: "dc11cd91-f5b5-45d6-8504-2d56367411a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:50.347496 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.347447 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc11cd91-f5b5-45d6-8504-2d56367411a9-kube-api-access-r2qfl\" (UniqueName: \"kubernetes.io/projected/dc11cd91-f5b5-45d6-8504-2d56367411a9-kube-api-access-r2qfl\") pod \"dc11cd91-f5b5-45d6-8504-2d56367411a9\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " Oct 29 23:06:50.347585 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.347518 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-metrics-certs\") pod \"dc11cd91-f5b5-45d6-8504-2d56367411a9\" (UID: \"dc11cd91-f5b5-45d6-8504-2d56367411a9\") " Oct 29 23:06:50.347933 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.347882 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11cd91-f5b5-45d6-8504-2d56367411a9-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.352523 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.352495 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-metrics-certs" pod "dc11cd91-f5b5-45d6-8504-2d56367411a9" (UID: "dc11cd91-f5b5-45d6-8504-2d56367411a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:50.367555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.364599 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-maas-api-tls" pod "dc11cd91-f5b5-45d6-8504-2d56367411a9" (UID: "dc11cd91-f5b5-45d6-8504-2d56367411a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:06:50.367555 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.366835 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc11cd91-f5b5-45d6-8504-2d56367411a9-kube-api-access-r2qfl" pod "dc11cd91-f5b5-45d6-8504-2d56367411a9" (UID: "dc11cd91-f5b5-45d6-8504-2d56367411a9"). InnerVolumeSpecName "kube-api-access-r2qfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:50.448569 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.448535 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qfl\" (UniqueName: \"kubernetes.io/projected/dc11cd91-f5b5-45d6-8504-2d56367411a9-kube-api-access-r2qfl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.448674 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.448574 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.448674 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.448588 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11cd91-f5b5-45d6-8504-2d56367411a9-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.568690 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.568646 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:50.702524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.702497 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 23:06:50.740788 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.740751 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:50.750790 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.750762 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-socket\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.750916 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.750823 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-token\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-token\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.750916 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.750889 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-envoy\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.751019 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.750930 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-podinfo\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.751019 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.750949 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-socket" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.751019 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.750980 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-credential-socket\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.751019 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751015 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-data\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.751227 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751046 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-certs\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.751227 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751076 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istiod-ca-cert\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.751227 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751172 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-kube-api-access-dkf74\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-kube-api-access-dkf74\") pod \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\" (UID: \"0c21b381-6e1d-4f1c-9234-554b1f4e0952\") " Oct 29 23:06:50.751422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751341 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-credential-socket" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.751485 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751427 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-certs" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.751848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751637 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-data" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.751848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751634 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istiod-ca-cert" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:06:50.751848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751670 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.751848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751694 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.751848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.751714 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.753680 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.753657 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-envoy" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:06:50.753836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.753817 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-token" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:50.754152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.754127 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-podinfo" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:06:50.754263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.754244 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-kube-api-access-dkf74" pod "0c21b381-6e1d-4f1c-9234-554b1f4e0952" (UID: "0c21b381-6e1d-4f1c-9234-554b1f4e0952"). InnerVolumeSpecName "kube-api-access-dkf74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:06:50.852863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.852776 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkf74\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-kube-api-access-dkf74\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.852863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.852809 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.852863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.852818 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.852863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.852826 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.852863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.852833 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:50.852863 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:50.852841 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c21b381-6e1d-4f1c-9234-554b1f4e0952-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:06:51.136590 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.136499 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" Oct 29 23:06:51.136590 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.136515 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts" event={"ID":"dc11cd91-f5b5-45d6-8504-2d56367411a9","Type":"ContainerDied","Data":"279f02a0bb8b3701b5df5b62039a79eac4beaef3dc765c7dafc00d2011a4334f"} Oct 29 23:06:51.136590 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.136586 5610 scope.go:122] "RemoveContainer" containerID="7fe6e5fc0a1bb2b6d2e5750bf7ebdd52835e5591917c6b74a0dcf2d3205871fc" Oct 29 23:06:51.137945 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.137917 5610 generic.go:356] "Generic (PLEG): container finished" podID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerID="f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b" exitCode=0 Oct 29 23:06:51.138048 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.137968 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" Oct 29 23:06:51.138048 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.138020 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" event={"ID":"0c21b381-6e1d-4f1c-9234-554b1f4e0952","Type":"ContainerDied","Data":"f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b"} Oct 29 23:06:51.138140 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.138063 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9" event={"ID":"0c21b381-6e1d-4f1c-9234-554b1f4e0952","Type":"ContainerDied","Data":"0610d4b0042fe0fe5cf93ecacb9808711ef1593b7ae2b45f03578262f4fd2528"} Oct 29 23:06:51.145193 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.145175 5610 scope.go:122] "RemoveContainer" containerID="f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b" Oct 29 23:06:51.155089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.155066 5610 scope.go:122] "RemoveContainer" containerID="f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b" Oct 29 23:06:51.155363 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:06:51.155341 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b\": container with ID starting with f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b not found: ID does not exist" containerID="f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b" Oct 29 23:06:51.155431 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.155374 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b"} err="failed to get container status \"f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b\": rpc error: code = NotFound desc = could not find container \"f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b\": container with ID starting with f809548856f065b5e9feef8cf7d60d4591ffeed3a5ee3ca49f63d4a28c8f545b not found: ID does not exist" Oct 29 23:06:51.209927 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.209895 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9"] Oct 29 23:06:51.222002 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.221970 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2ae845-openshift-default-7bb6f6fdcf-ftbp9"] Oct 29 23:06:51.243607 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.243572 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:51.274114 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.274083 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts"] Oct 29 23:06:51.277531 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.277505 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-92fd86-67d6c6d57c-p4bts"] Oct 29 23:06:51.412891 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.412793 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:51.634529 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.634497 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" path="/var/lib/kubelet/pods/0c21b381-6e1d-4f1c-9234-554b1f4e0952/volumes" Oct 29 23:06:51.634938 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:51.634924 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc11cd91-f5b5-45d6-8504-2d56367411a9" path="/var/lib/kubelet/pods/dc11cd91-f5b5-45d6-8504-2d56367411a9/volumes" Oct 29 23:06:52.519865 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:52.519832 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:54.089766 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:54.089726 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:55.490758 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:55.490712 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:55.649096 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:55.649061 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:55.723480 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:55.723440 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5"] Oct 29 23:06:55.723686 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:55.723662 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" containerID="cri-o://d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9" gracePeriod=30 Oct 29 23:06:55.806861 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:55.806770 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:56.264843 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:56.264805 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:56.617451 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:56.617341 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" podUID="12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:57.014679 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:57.014639 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw"] Oct 29 23:06:57.170306 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:57.170263 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:57.308135 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:57.308042 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:57.629081 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:57.628987 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" podUID="7cb82a55-445c-40e7-95b3-a40ba0688d71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:06:58.007980 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:58.007897 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:58.156173 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:58.156136 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:58.257531 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:58.257500 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff"] Oct 29 23:06:58.257751 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:58.257721 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" containerID="cri-o://bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241" gracePeriod=30 Oct 29 23:06:58.338438 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:58.338400 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:58.740781 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:58.740741 5610 patch_prober.go:29] interesting pod/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:58.741182 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:58.740804 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:06:58.845724 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:58.845682 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:06:59.321114 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:59.321073 5610 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 29 23:06:59.321327 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:06:59.321141 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 29 23:07:00.139729 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.139700 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 23:07:00.182462 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.182367 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 23:07:00.189121 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.189091 5610 generic.go:356] "Generic (PLEG): container finished" podID="7cb82a55-445c-40e7-95b3-a40ba0688d71" containerID="926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee" exitCode=0 Oct 29 23:07:00.189282 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.189147 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" Oct 29 23:07:00.189282 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.189169 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" event={"ID":"7cb82a55-445c-40e7-95b3-a40ba0688d71","Type":"ContainerDied","Data":"926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee"} Oct 29 23:07:00.189282 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.189251 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd" event={"ID":"7cb82a55-445c-40e7-95b3-a40ba0688d71","Type":"ContainerDied","Data":"af556123a8ae5f4d8b9b31d754fe559fd4986d313564982d5d4bceace0bf63a8"} Oct 29 23:07:00.189282 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.189273 5610 scope.go:122] "RemoveContainer" containerID="926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee" Oct 29 23:07:00.190658 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.190634 5610 generic.go:356] "Generic (PLEG): container finished" podID="12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" containerID="5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab" exitCode=0 Oct 29 23:07:00.190779 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.190686 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" Oct 29 23:07:00.190779 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.190711 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" event={"ID":"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed","Type":"ContainerDied","Data":"5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab"} Oct 29 23:07:00.190779 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.190748 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn" event={"ID":"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed","Type":"ContainerDied","Data":"6aaa07a78be8932eb4116772785121d7ba0e37d595cdaf4cf94d923074535641"} Oct 29 23:07:00.199240 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.199217 5610 scope.go:122] "RemoveContainer" containerID="926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee" Oct 29 23:07:00.199558 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:00.199533 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee\": container with ID starting with 926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee not found: ID does not exist" containerID="926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee" Oct 29 23:07:00.199654 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.199566 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee"} err="failed to get container status \"926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee\": rpc error: code = NotFound desc = could not find container \"926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee\": container with ID starting with 926801486c036f7f7dad09ac973fc5e8fefb4114f82308350c791a6c960e55ee not found: ID does not exist" Oct 29 23:07:00.199654 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.199590 5610 scope.go:122] "RemoveContainer" containerID="5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab" Oct 29 23:07:00.209820 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.209802 5610 scope.go:122] "RemoveContainer" containerID="5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab" Oct 29 23:07:00.210103 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:00.210085 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab\": container with ID starting with 5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab not found: ID does not exist" containerID="5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab" Oct 29 23:07:00.210170 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.210124 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab"} err="failed to get container status \"5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab\": rpc error: code = NotFound desc = could not find container \"5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab\": container with ID starting with 5c19278636e5072219347a776f3e9c0e603c2b9e8f1568e817f2d13967688dab not found: ID does not exist" Oct 29 23:07:00.250315 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.250282 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb82a55-445c-40e7-95b3-a40ba0688d71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb82a55-445c-40e7-95b3-a40ba0688d71-plugins-config-volume\") pod \"7cb82a55-445c-40e7-95b3-a40ba0688d71\" (UID: \"7cb82a55-445c-40e7-95b3-a40ba0688d71\") " Oct 29 23:07:00.250530 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.250337 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb82a55-445c-40e7-95b3-a40ba0688d71-kube-api-access-z8k4t\" (UniqueName: \"kubernetes.io/projected/7cb82a55-445c-40e7-95b3-a40ba0688d71-kube-api-access-z8k4t\") pod \"7cb82a55-445c-40e7-95b3-a40ba0688d71\" (UID: \"7cb82a55-445c-40e7-95b3-a40ba0688d71\") " Oct 29 23:07:00.250530 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.250404 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-plugins-config-volume\") pod \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\" (UID: \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\") " Oct 29 23:07:00.250530 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.250424 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-kube-api-access-9mtfd\" (UniqueName: \"kubernetes.io/projected/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-kube-api-access-9mtfd\") pod \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\" (UID: \"12cd46c7-31eb-49f4-9ad6-59f83c8f2aed\") " Oct 29 23:07:00.250799 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.250770 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb82a55-445c-40e7-95b3-a40ba0688d71-plugins-config-volume" pod "7cb82a55-445c-40e7-95b3-a40ba0688d71" (UID: "7cb82a55-445c-40e7-95b3-a40ba0688d71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:00.250910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.250781 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-plugins-config-volume" pod "12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" (UID: "12cd46c7-31eb-49f4-9ad6-59f83c8f2aed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:00.252731 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.252705 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb82a55-445c-40e7-95b3-a40ba0688d71-kube-api-access-z8k4t" pod "7cb82a55-445c-40e7-95b3-a40ba0688d71" (UID: "7cb82a55-445c-40e7-95b3-a40ba0688d71"). InnerVolumeSpecName "kube-api-access-z8k4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:00.252922 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.252901 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-kube-api-access-9mtfd" pod "12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" (UID: "12cd46c7-31eb-49f4-9ad6-59f83c8f2aed"). InnerVolumeSpecName "kube-api-access-9mtfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:00.352044 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.351954 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cb82a55-445c-40e7-95b3-a40ba0688d71-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.352044 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.351984 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8k4t\" (UniqueName: \"kubernetes.io/projected/7cb82a55-445c-40e7-95b3-a40ba0688d71-kube-api-access-z8k4t\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.352044 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.351993 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.352044 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.352004 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtfd\" (UniqueName: \"kubernetes.io/projected/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed-kube-api-access-9mtfd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:00.444505 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.444462 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:00.536847 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.536807 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd"] Oct 29 23:07:00.545862 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.539650 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2ae845-5879bb5c87-fm2xd"] Oct 29 23:07:00.585134 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.585097 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn"] Oct 29 23:07:00.589751 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.589704 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2ae845-748796f66d-7m2rn"] Oct 29 23:07:00.979490 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:00.979450 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 23:07:01.057530 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057431 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-credential-socket\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.057530 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057479 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce818887-c1a6-452b-9584-432c17e64752-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce818887-c1a6-452b-9584-432c17e64752-istio-podinfo\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.057530 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057512 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-socket\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.057836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057571 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-kube-api-access-9lpmd\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-kube-api-access-9lpmd\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.057836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057594 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-data\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.057836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057656 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-envoy\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.057836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057678 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-credential-socket" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.057836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057696 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce818887-c1a6-452b-9584-432c17e64752-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce818887-c1a6-452b-9584-432c17e64752-istiod-ca-cert\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.057836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057728 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-istio-token\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-istio-token\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.057836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057761 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-certs\") pod \"ce818887-c1a6-452b-9584-432c17e64752\" (UID: \"ce818887-c1a6-452b-9584-432c17e64752\") " Oct 29 23:07:01.058238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.057898 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-data" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.058238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.058084 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-certs" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.058238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.058157 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.058238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.058162 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce818887-c1a6-452b-9584-432c17e64752-istiod-ca-cert" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:01.058238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.058174 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.058238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.058206 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.058619 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.058256 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-socket" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.060016 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.059992 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-istio-token" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.060162 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.060140 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-envoy" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:01.060235 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.060176 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce818887-c1a6-452b-9584-432c17e64752-istio-podinfo" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:01.060294 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.060265 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-kube-api-access-9lpmd" pod "ce818887-c1a6-452b-9584-432c17e64752" (UID: "ce818887-c1a6-452b-9584-432c17e64752"). InnerVolumeSpecName "kube-api-access-9lpmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:01.159299 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.159263 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.159299 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.159295 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce818887-c1a6-452b-9584-432c17e64752-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.159299 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.159304 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.159755 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.159313 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce818887-c1a6-452b-9584-432c17e64752-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.159755 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.159324 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce818887-c1a6-452b-9584-432c17e64752-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.159755 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.159333 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lpmd\" (UniqueName: \"kubernetes.io/projected/ce818887-c1a6-452b-9584-432c17e64752-kube-api-access-9lpmd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:01.199859 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.199821 5610 generic.go:356] "Generic (PLEG): container finished" podID="ce818887-c1a6-452b-9584-432c17e64752" containerID="d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9" exitCode=0 Oct 29 23:07:01.200049 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.199898 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" event={"ID":"ce818887-c1a6-452b-9584-432c17e64752","Type":"ContainerDied","Data":"d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9"} Oct 29 23:07:01.200049 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.199961 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" event={"ID":"ce818887-c1a6-452b-9584-432c17e64752","Type":"ContainerDied","Data":"eaff0b3c791184356053c3079677a909463b23071d32389577f066f93851cd3f"} Oct 29 23:07:01.200049 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.199908 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5" Oct 29 23:07:01.200049 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.199988 5610 scope.go:122] "RemoveContainer" containerID="d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9" Oct 29 23:07:01.211072 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.211039 5610 scope.go:122] "RemoveContainer" containerID="f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0" Oct 29 23:07:01.221630 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.221604 5610 scope.go:122] "RemoveContainer" containerID="d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9" Oct 29 23:07:01.221908 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:01.221888 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9\": container with ID starting with d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9 not found: ID does not exist" containerID="d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9" Oct 29 23:07:01.221980 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.221916 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9"} err="failed to get container status \"d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9\": rpc error: code = NotFound desc = could not find container \"d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9\": container with ID starting with d8bab46c4b23cb3e393fcdfb2777522d6a488425528f0680103209b485bfdbc9 not found: ID does not exist" Oct 29 23:07:01.221980 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.221934 5610 scope.go:122] "RemoveContainer" containerID="f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0" Oct 29 23:07:01.222185 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:01.222164 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0\": container with ID starting with f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0 not found: ID does not exist" containerID="f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0" Oct 29 23:07:01.222251 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.222191 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0"} err="failed to get container status \"f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0\": rpc error: code = NotFound desc = could not find container \"f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0\": container with ID starting with f22df46d03f79a8948c80c904c01df472efc6c5cc8e0830b5ec6602799cf1cc0 not found: ID does not exist" Oct 29 23:07:01.255281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.255245 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5"] Oct 29 23:07:01.258898 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.258870 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8f1627-openshift-default-6dc95df59f-dncl5"] Oct 29 23:07:01.353096 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.352975 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:01.634598 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.634500 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" path="/var/lib/kubelet/pods/12cd46c7-31eb-49f4-9ad6-59f83c8f2aed/volumes" Oct 29 23:07:01.634902 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.634887 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb82a55-445c-40e7-95b3-a40ba0688d71" path="/var/lib/kubelet/pods/7cb82a55-445c-40e7-95b3-a40ba0688d71/volumes" Oct 29 23:07:01.635293 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:01.635281 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce818887-c1a6-452b-9584-432c17e64752" path="/var/lib/kubelet/pods/ce818887-c1a6-452b-9584-432c17e64752/volumes" Oct 29 23:07:02.015578 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:02.015530 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" podUID="d4751898-78dc-401d-834b-d5bfb1dca7ff" containerName="payload-pre-processing" containerID="cri-o://bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979" gracePeriod=25 Oct 29 23:07:02.051959 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:02.051924 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:03.515329 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.515300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:03.534890 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.534820 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 23:07:03.684820 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.684781 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-token\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-token\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.684839 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-kube-api-access-vfj4g\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-kube-api-access-vfj4g\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.684875 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-credential-socket\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.684908 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-podinfo\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.684944 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-socket\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685250 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685027 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-data\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685250 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685095 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istiod-ca-cert\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685250 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685124 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-certs\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685250 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685205 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-envoy\") pod \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\" (UID: \"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a\") " Oct 29 23:07:03.685491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685337 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-socket" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.685558 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685497 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-data" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.685639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685609 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istiod-ca-cert" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:03.685771 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685747 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-certs" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.685848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685771 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-credential-socket" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.685848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685809 5610 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.685848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.685838 5610 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.687164 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.687141 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-kube-api-access-vfj4g" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "kube-api-access-vfj4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:03.687340 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.687312 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-token" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:03.687527 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.687508 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-envoy" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:07:03.687738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.687721 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-podinfo" pod "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" (UID: "fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 29 23:07:03.786584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.786503 5610 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.786584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.786531 5610 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.786584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.786540 5610 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.786584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.786548 5610 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.786584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.786556 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfj4g\" (UniqueName: \"kubernetes.io/projected/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-kube-api-access-vfj4g\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.786584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.786566 5610 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:03.786584 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:03.786575 5610 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:04.218541 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.218505 5610 generic.go:356] "Generic (PLEG): container finished" podID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerID="bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241" exitCode=0 Oct 29 23:07:04.218789 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.218575 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" Oct 29 23:07:04.218789 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.218596 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" event={"ID":"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a","Type":"ContainerDied","Data":"bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241"} Oct 29 23:07:04.218789 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.218659 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff" event={"ID":"fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a","Type":"ContainerDied","Data":"eceac7cb69b94c6dfbf53b6f5195b45ec477fc45371928c28836f4f769c75a81"} Oct 29 23:07:04.218789 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.218681 5610 scope.go:122] "RemoveContainer" containerID="bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241" Oct 29 23:07:04.228822 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.228804 5610 scope.go:122] "RemoveContainer" containerID="bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241" Oct 29 23:07:04.229084 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:04.229061 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241\": container with ID starting with bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241 not found: ID does not exist" containerID="bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241" Oct 29 23:07:04.229160 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.229099 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241"} err="failed to get container status \"bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241\": rpc error: code = NotFound desc = could not find container \"bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241\": container with ID starting with bfad4acde9ef27c5e45796f684905e24400be0e0a0280ff0beb3392ffba9c241 not found: ID does not exist" Oct 29 23:07:04.279474 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.279441 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff"] Oct 29 23:07:04.291362 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.291335 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-92fd86-openshift-default-5c74588f5-79kff"] Oct 29 23:07:04.807060 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.807025 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" podUID="d4751898-78dc-401d-834b-d5bfb1dca7ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:04.850058 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:04.850009 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:05.642045 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:05.642008 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" path="/var/lib/kubelet/pods/fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a/volumes" Oct 29 23:07:06.341427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:06.341355 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:07.028040 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:07.028005 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:07.166640 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:07.166607 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:08.513992 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:08.513952 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:08.705190 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:08.705148 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:08.957309 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:08.957278 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:09.435325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:09.435233 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:10.210015 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:10.209975 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:12.705514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:12.705481 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:13.105812 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:13.105708 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:13.251374 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:13.251336 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:13.711243 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:13.711208 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:07:13.735263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:13.735229 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:14.807671 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:14.807627 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" podUID="d4751898-78dc-401d-834b-d5bfb1dca7ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:07:15.778334 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:15.778272 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:19.749216 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:19.749182 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:19.894744 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:19.894701 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:20.133452 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.133337 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:20.161122 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.161079 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz"] Oct 29 23:07:20.161439 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.161407 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" podUID="eacc6253-bd0f-4197-b8c0-eab3a4732971" containerName="maas-api" containerID="cri-o://2c94420a6caa2e8f56548fa70995eaddd3fd7e9832c49759b544cd6bed52a7f0" gracePeriod=30 Oct 29 23:07:20.284202 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.284171 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:20.301253 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.301214 5610 generic.go:356] "Generic (PLEG): container finished" podID="eacc6253-bd0f-4197-b8c0-eab3a4732971" containerID="2c94420a6caa2e8f56548fa70995eaddd3fd7e9832c49759b544cd6bed52a7f0" exitCode=0 Oct 29 23:07:20.301438 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.301276 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" event={"ID":"eacc6253-bd0f-4197-b8c0-eab3a4732971","Type":"ContainerDied","Data":"2c94420a6caa2e8f56548fa70995eaddd3fd7e9832c49759b544cd6bed52a7f0"} Oct 29 23:07:20.420921 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.420892 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 23:07:20.445864 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.445838 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eacc6253-bd0f-4197-b8c0-eab3a4732971-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc6253-bd0f-4197-b8c0-eab3a4732971-openshift-service-ca\") pod \"eacc6253-bd0f-4197-b8c0-eab3a4732971\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " Oct 29 23:07:20.446033 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.445895 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eacc6253-bd0f-4197-b8c0-eab3a4732971-kube-api-access-mvkfs\" (UniqueName: \"kubernetes.io/projected/eacc6253-bd0f-4197-b8c0-eab3a4732971-kube-api-access-mvkfs\") pod \"eacc6253-bd0f-4197-b8c0-eab3a4732971\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " Oct 29 23:07:20.446033 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.445939 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-metrics-certs\") pod \"eacc6253-bd0f-4197-b8c0-eab3a4732971\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " Oct 29 23:07:20.446033 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.445964 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls\") pod \"eacc6253-bd0f-4197-b8c0-eab3a4732971\" (UID: \"eacc6253-bd0f-4197-b8c0-eab3a4732971\") " Oct 29 23:07:20.446231 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.446180 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacc6253-bd0f-4197-b8c0-eab3a4732971-openshift-service-ca" pod "eacc6253-bd0f-4197-b8c0-eab3a4732971" (UID: "eacc6253-bd0f-4197-b8c0-eab3a4732971"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:20.446297 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.446275 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc6253-bd0f-4197-b8c0-eab3a4732971-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.448001 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.447970 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eacc6253-bd0f-4197-b8c0-eab3a4732971-kube-api-access-mvkfs" pod "eacc6253-bd0f-4197-b8c0-eab3a4732971" (UID: "eacc6253-bd0f-4197-b8c0-eab3a4732971"). InnerVolumeSpecName "kube-api-access-mvkfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:20.448111 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.448013 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls" pod "eacc6253-bd0f-4197-b8c0-eab3a4732971" (UID: "eacc6253-bd0f-4197-b8c0-eab3a4732971"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:20.448111 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.448052 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-metrics-certs" pod "eacc6253-bd0f-4197-b8c0-eab3a4732971" (UID: "eacc6253-bd0f-4197-b8c0-eab3a4732971"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:20.546918 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.546795 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvkfs\" (UniqueName: \"kubernetes.io/projected/eacc6253-bd0f-4197-b8c0-eab3a4732971-kube-api-access-mvkfs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.546918 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.546830 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.546918 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.546840 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eacc6253-bd0f-4197-b8c0-eab3a4732971-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:20.674489 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.674383 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:20.835465 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:20.835429 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:21.280773 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:21.280731 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:21.317431 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:21.315587 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" event={"ID":"eacc6253-bd0f-4197-b8c0-eab3a4732971","Type":"ContainerDied","Data":"ff9e09649cd061fbcbb32b505e129e90bd8a773167d67349b8e63d0f5c527155"} Oct 29 23:07:21.317431 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:21.315661 5610 scope.go:122] "RemoveContainer" containerID="2c94420a6caa2e8f56548fa70995eaddd3fd7e9832c49759b544cd6bed52a7f0" Oct 29 23:07:21.317431 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:21.315818 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz" Oct 29 23:07:21.365177 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:21.365090 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz"] Oct 29 23:07:21.376808 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:21.376769 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-95672c-685684857c-xt2bz"] Oct 29 23:07:21.634987 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:21.634883 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eacc6253-bd0f-4197-b8c0-eab3a4732971" path="/var/lib/kubelet/pods/eacc6253-bd0f-4197-b8c0-eab3a4732971/volumes" Oct 29 23:07:22.286564 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.286501 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 23:07:22.324957 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.324919 5610 generic.go:356] "Generic (PLEG): container finished" podID="d4751898-78dc-401d-834b-d5bfb1dca7ff" containerID="bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979" exitCode=0 Oct 29 23:07:22.325208 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.324968 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" Oct 29 23:07:22.325208 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.325010 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" event={"ID":"d4751898-78dc-401d-834b-d5bfb1dca7ff","Type":"ContainerDied","Data":"bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979"} Oct 29 23:07:22.325208 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.325053 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw" event={"ID":"d4751898-78dc-401d-834b-d5bfb1dca7ff","Type":"ContainerDied","Data":"72a43e9d29f8bdc9d450c4031bd5b40f9f3c9b87137187da3b6f968e8fc891ae"} Oct 29 23:07:22.325208 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.325082 5610 scope.go:122] "RemoveContainer" containerID="bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979" Oct 29 23:07:22.335232 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.335208 5610 scope.go:122] "RemoveContainer" containerID="bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979" Oct 29 23:07:22.335616 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:22.335585 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979\": container with ID starting with bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979 not found: ID does not exist" containerID="bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979" Oct 29 23:07:22.335741 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.335627 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979"} err="failed to get container status \"bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979\": rpc error: code = NotFound desc = could not find container \"bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979\": container with ID starting with bfa425cb2002739ab0f3b4ddb17b5c88e9997faeeb797b04755cfaa7cc96e979 not found: ID does not exist" Oct 29 23:07:22.358853 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.358825 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4751898-78dc-401d-834b-d5bfb1dca7ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4751898-78dc-401d-834b-d5bfb1dca7ff-plugins-config-volume\") pod \"d4751898-78dc-401d-834b-d5bfb1dca7ff\" (UID: \"d4751898-78dc-401d-834b-d5bfb1dca7ff\") " Oct 29 23:07:22.358853 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.358861 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4751898-78dc-401d-834b-d5bfb1dca7ff-kube-api-access-mt9vh\" (UniqueName: \"kubernetes.io/projected/d4751898-78dc-401d-834b-d5bfb1dca7ff-kube-api-access-mt9vh\") pod \"d4751898-78dc-401d-834b-d5bfb1dca7ff\" (UID: \"d4751898-78dc-401d-834b-d5bfb1dca7ff\") " Oct 29 23:07:22.359308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.359280 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4751898-78dc-401d-834b-d5bfb1dca7ff-plugins-config-volume" pod "d4751898-78dc-401d-834b-d5bfb1dca7ff" (UID: "d4751898-78dc-401d-834b-d5bfb1dca7ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:07:22.361077 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.361052 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4751898-78dc-401d-834b-d5bfb1dca7ff-kube-api-access-mt9vh" pod "d4751898-78dc-401d-834b-d5bfb1dca7ff" (UID: "d4751898-78dc-401d-834b-d5bfb1dca7ff"). InnerVolumeSpecName "kube-api-access-mt9vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:22.459677 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.459641 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4751898-78dc-401d-834b-d5bfb1dca7ff-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:22.459677 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.459672 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt9vh\" (UniqueName: \"kubernetes.io/projected/d4751898-78dc-401d-834b-d5bfb1dca7ff-kube-api-access-mt9vh\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:22.668816 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.668782 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw"] Oct 29 23:07:22.672182 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:22.672157 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4a0287-7c4d7dff9f-vf2cw"] Oct 29 23:07:23.634740 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:23.634704 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4751898-78dc-401d-834b-d5bfb1dca7ff" path="/var/lib/kubelet/pods/d4751898-78dc-401d-834b-d5bfb1dca7ff/volumes" Oct 29 23:07:27.730936 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:27.730890 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:27.911688 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:27.911648 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:28.089617 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:28.089520 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:28.261302 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:28.261266 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:29.855127 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:29.855093 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:30.768629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:30.768591 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86789cb564-phf64"] Oct 29 23:07:31.373287 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.373132 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-86789cb564-phf64" podUID="f7f71636-1271-4e26-b3a4-cad6448fb72a" containerName="manager" containerID="cri-o://0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140" gracePeriod=30 Oct 29 23:07:31.636051 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.635987 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 23:07:31.749706 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.749671 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-metrics-certs\") pod \"f7f71636-1271-4e26-b3a4-cad6448fb72a\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " Oct 29 23:07:31.749926 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.749723 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-webhook-cert\") pod \"f7f71636-1271-4e26-b3a4-cad6448fb72a\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " Oct 29 23:07:31.749926 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.749760 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f71636-1271-4e26-b3a4-cad6448fb72a-kube-api-access-5bt8g\" (UniqueName: \"kubernetes.io/projected/f7f71636-1271-4e26-b3a4-cad6448fb72a-kube-api-access-5bt8g\") pod \"f7f71636-1271-4e26-b3a4-cad6448fb72a\" (UID: \"f7f71636-1271-4e26-b3a4-cad6448fb72a\") " Oct 29 23:07:31.751768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.751736 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f71636-1271-4e26-b3a4-cad6448fb72a-kube-api-access-5bt8g" pod "f7f71636-1271-4e26-b3a4-cad6448fb72a" (UID: "f7f71636-1271-4e26-b3a4-cad6448fb72a"). InnerVolumeSpecName "kube-api-access-5bt8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:07:31.751894 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.751792 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-metrics-certs" pod "f7f71636-1271-4e26-b3a4-cad6448fb72a" (UID: "f7f71636-1271-4e26-b3a4-cad6448fb72a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:31.751894 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.751849 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-webhook-cert" pod "f7f71636-1271-4e26-b3a4-cad6448fb72a" (UID: "f7f71636-1271-4e26-b3a4-cad6448fb72a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:07:31.850783 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.850740 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:31.850783 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.850772 5610 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f7f71636-1271-4e26-b3a4-cad6448fb72a-webhook-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:31.850783 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:31.850781 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bt8g\" (UniqueName: \"kubernetes.io/projected/f7f71636-1271-4e26-b3a4-cad6448fb72a-kube-api-access-5bt8g\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:07:32.379747 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.379712 5610 generic.go:356] "Generic (PLEG): container finished" podID="f7f71636-1271-4e26-b3a4-cad6448fb72a" containerID="0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140" exitCode=0 Oct 29 23:07:32.380262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.379754 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-phf64" Oct 29 23:07:32.380262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.379790 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-phf64" event={"ID":"f7f71636-1271-4e26-b3a4-cad6448fb72a","Type":"ContainerDied","Data":"0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140"} Oct 29 23:07:32.380262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.379845 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-phf64" event={"ID":"f7f71636-1271-4e26-b3a4-cad6448fb72a","Type":"ContainerDied","Data":"456c1eb390e8aada7bcb978c4892bbe9c90513c2f1b9c182ba00165414179c28"} Oct 29 23:07:32.380262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.379870 5610 scope.go:122] "RemoveContainer" containerID="0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140" Oct 29 23:07:32.391145 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.391120 5610 scope.go:122] "RemoveContainer" containerID="0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140" Oct 29 23:07:32.391468 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:32.391443 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140\": container with ID starting with 0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140 not found: ID does not exist" containerID="0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140" Oct 29 23:07:32.391566 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.391479 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140"} err="failed to get container status \"0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140\": rpc error: code = NotFound desc = could not find container \"0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140\": container with ID starting with 0449b71d553f024a2f35b4b22d98efa490f4884180d51f485d4e289cc9708140 not found: ID does not exist" Oct 29 23:07:32.428167 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.428132 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86789cb564-phf64"] Oct 29 23:07:32.440654 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:32.440624 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86789cb564-phf64"] Oct 29 23:07:33.117623 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.117585 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86789cb564-4vgtn"] Oct 29 23:07:33.118169 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118148 5610 cpu_manager.go:414] "containerMap: removing container" podUID="caeab242-8754-4bee-8f24-41b26ed86f2d" containerName="maas-api" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118170 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caeab242-8754-4bee-8f24-41b26ed86f2d" containerName="maas-api" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118188 5610 cpu_manager.go:414] "containerMap: removing container" podUID="12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" containerName="payload-processing" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118196 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" containerName="payload-processing" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118206 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3a59f3f1-c569-4d63-9896-42d459c6f7bd" containerName="payload-pre-processing" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118213 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a59f3f1-c569-4d63-9896-42d459c6f7bd" containerName="payload-pre-processing" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118235 5610 cpu_manager.go:414] "containerMap: removing container" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118242 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118253 5610 cpu_manager.go:414] "containerMap: removing container" podUID="dc11cd91-f5b5-45d6-8504-2d56367411a9" containerName="maas-api" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118260 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc11cd91-f5b5-45d6-8504-2d56367411a9" containerName="maas-api" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118270 5610 cpu_manager.go:414] "containerMap: removing container" podUID="80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" containerName="maas-api" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118277 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" containerName="maas-api" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118288 5610 cpu_manager.go:414] "containerMap: removing container" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" Oct 29 23:07:33.118289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118295 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118307 5610 cpu_manager.go:414] "containerMap: removing container" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118314 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118328 5610 cpu_manager.go:414] "containerMap: removing container" podUID="460af1df-6d95-4d73-923a-61bd9ba7f11a" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118336 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460af1df-6d95-4d73-923a-61bd9ba7f11a" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118349 5610 cpu_manager.go:414] "containerMap: removing container" podUID="82fb649c-6c0b-456c-ae9b-c3e52195fc53" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118356 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fb649c-6c0b-456c-ae9b-c3e52195fc53" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118367 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118374 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118387 5610 cpu_manager.go:414] "containerMap: removing container" podUID="86dd2ee8-782a-4af7-bc3a-d5afc675e88f" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118406 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86dd2ee8-782a-4af7-bc3a-d5afc675e88f" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118419 5610 cpu_manager.go:414] "containerMap: removing container" podUID="dcfc3c40-8c05-4bce-885f-424f8742c844" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118426 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfc3c40-8c05-4bce-885f-424f8742c844" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118436 5610 cpu_manager.go:414] "containerMap: removing container" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118443 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118452 5610 cpu_manager.go:414] "containerMap: removing container" podUID="2eee6ddd-7369-4268-b2fb-74c11ed01e3d" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118459 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eee6ddd-7369-4268-b2fb-74c11ed01e3d" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118471 5610 cpu_manager.go:414] "containerMap: removing container" podUID="60c6778f-8a94-4a33-bc9e-2ed413fd0d70" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118478 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c6778f-8a94-4a33-bc9e-2ed413fd0d70" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118492 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f7f71636-1271-4e26-b3a4-cad6448fb72a" containerName="manager" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118498 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f71636-1271-4e26-b3a4-cad6448fb72a" containerName="manager" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118511 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3b832c48-65fb-4aa1-b3fe-80e242f6e134" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118519 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b832c48-65fb-4aa1-b3fe-80e242f6e134" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118529 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3a68a161-97bd-429d-b6bb-28786285b6a9" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118536 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a68a161-97bd-429d-b6bb-28786285b6a9" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118547 5610 cpu_manager.go:414] "containerMap: removing container" podUID="56496049-8920-4b5f-b18d-7d536f38feb0" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118555 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56496049-8920-4b5f-b18d-7d536f38feb0" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118570 5610 cpu_manager.go:414] "containerMap: removing container" podUID="43d156d3-b215-4564-9052-db75adf96095" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118578 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d156d3-b215-4564-9052-db75adf96095" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118588 5610 cpu_manager.go:414] "containerMap: removing container" podUID="3331beb8-c6f0-4036-a335-93ca6648616f" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118595 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3331beb8-c6f0-4036-a335-93ca6648616f" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118608 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d4751898-78dc-401d-834b-d5bfb1dca7ff" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118615 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4751898-78dc-401d-834b-d5bfb1dca7ff" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118626 5610 cpu_manager.go:414] "containerMap: removing container" podUID="896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118633 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118645 5610 cpu_manager.go:414] "containerMap: removing container" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118653 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118664 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118671 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118684 5610 cpu_manager.go:414] "containerMap: removing container" podUID="1ff95e20-7adf-4a33-821d-1d51c58e0d58" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118691 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff95e20-7adf-4a33-821d-1d51c58e0d58" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118705 5610 cpu_manager.go:414] "containerMap: removing container" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118712 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118721 5610 cpu_manager.go:414] "containerMap: removing container" podUID="7cb82a55-445c-40e7-95b3-a40ba0688d71" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118728 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb82a55-445c-40e7-95b3-a40ba0688d71" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118739 5610 cpu_manager.go:414] "containerMap: removing container" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118746 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118756 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118780 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118795 5610 cpu_manager.go:414] "containerMap: removing container" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118804 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118814 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118821 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118834 5610 cpu_manager.go:414] "containerMap: removing container" podUID="65038ec5-df43-48fa-b380-d02bc2ec4a83" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118841 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65038ec5-df43-48fa-b380-d02bc2ec4a83" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118855 5610 cpu_manager.go:414] "containerMap: removing container" podUID="38e06ae3-0c7d-4158-9e15-63e75c718dc0" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118862 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e06ae3-0c7d-4158-9e15-63e75c718dc0" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118873 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ce072c8e-2cca-4616-916d-2d2ddb52d113" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118882 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce072c8e-2cca-4616-916d-2d2ddb52d113" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118892 5610 cpu_manager.go:414] "containerMap: removing container" podUID="f7d38d15-b38f-423b-aeef-68f36963738c" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118899 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d38d15-b38f-423b-aeef-68f36963738c" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118909 5610 cpu_manager.go:414] "containerMap: removing container" podUID="cdc1edc1-003a-4e57-9ab6-8c789418cb15" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118916 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc1edc1-003a-4e57-9ab6-8c789418cb15" containerName="payload-pre-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118925 5610 cpu_manager.go:414] "containerMap: removing container" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118932 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118949 5610 cpu_manager.go:414] "containerMap: removing container" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118956 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118977 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118985 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.118999 5610 cpu_manager.go:414] "containerMap: removing container" podUID="eacc6253-bd0f-4197-b8c0-eab3a4732971" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119007 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eacc6253-bd0f-4197-b8c0-eab3a4732971" containerName="maas-api" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119017 5610 cpu_manager.go:414] "containerMap: removing container" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119024 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119033 5610 cpu_manager.go:414] "containerMap: removing container" podUID="91a0bb3e-f76d-4030-a53c-ac5905587c93" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119040 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a0bb3e-f76d-4030-a53c-ac5905587c93" containerName="payload-processing" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119136 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" Oct 29 23:07:33.119106 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119152 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119162 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="eacc6253-bd0f-4197-b8c0-eab3a4732971" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119170 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119177 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a0bb3e-f76d-4030-a53c-ac5905587c93" containerName="payload-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119187 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="caeab242-8754-4bee-8f24-41b26ed86f2d" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119198 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="12cd46c7-31eb-49f4-9ad6-59f83c8f2aed" containerName="payload-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119207 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a59f3f1-c569-4d63-9896-42d459c6f7bd" containerName="payload-pre-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119217 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119225 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc11cd91-f5b5-45d6-8504-2d56367411a9" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119233 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e8c33c-39e3-4aaa-a59a-4d78a7e6cc6a" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119243 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a39cdad-2f97-46b0-bf8b-9910a2963607" containerName="istio-proxy" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119252 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c21b381-6e1d-4f1c-9234-554b1f4e0952" containerName="istio-proxy" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119261 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="460af1df-6d95-4d73-923a-61bd9ba7f11a" containerName="payload-pre-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119272 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fb649c-6c0b-456c-ae9b-c3e52195fc53" containerName="payload-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119282 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119291 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="86dd2ee8-782a-4af7-bc3a-d5afc675e88f" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119299 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfc3c40-8c05-4bce-885f-424f8742c844" containerName="payload-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119307 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdcd015-f56a-4206-b8cb-d9fe6cfbb68a" containerName="istio-proxy" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119315 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eee6ddd-7369-4268-b2fb-74c11ed01e3d" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119323 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c6778f-8a94-4a33-bc9e-2ed413fd0d70" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119332 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f71636-1271-4e26-b3a4-cad6448fb72a" containerName="manager" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119342 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a68a161-97bd-429d-b6bb-28786285b6a9" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119350 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="56496049-8920-4b5f-b18d-7d536f38feb0" containerName="payload-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119359 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b832c48-65fb-4aa1-b3fe-80e242f6e134" containerName="payload-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119368 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d156d3-b215-4564-9052-db75adf96095" containerName="payload-pre-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119377 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="3331beb8-c6f0-4036-a335-93ca6648616f" containerName="payload-pre-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119387 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4751898-78dc-401d-834b-d5bfb1dca7ff" containerName="payload-pre-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119412 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="896bef94-faf6-4ff5-b7b6-4c3dc5d09b3a" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119419 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0a67a0-a252-4a00-94c5-cc659b9a877b" containerName="main" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119429 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2d8c3e-e6b1-47a5-a26d-1d923d92a685" containerName="istio-proxy" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119438 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff95e20-7adf-4a33-821d-1d51c58e0d58" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119447 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8052bf-194c-495c-a1d4-4431b30768ce" containerName="main" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119456 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb82a55-445c-40e7-95b3-a40ba0688d71" containerName="payload-pre-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119464 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05ee9f3-b2e7-49ac-a2d8-a142d73466b4" containerName="payload-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119475 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="65038ec5-df43-48fa-b380-d02bc2ec4a83" containerName="payload-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119485 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e06ae3-0c7d-4158-9e15-63e75c718dc0" containerName="payload-pre-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119494 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce072c8e-2cca-4616-916d-2d2ddb52d113" containerName="payload-pre-processing" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119501 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d38d15-b38f-423b-aeef-68f36963738c" containerName="maas-api" Oct 29 23:07:33.122168 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.119510 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc1edc1-003a-4e57-9ab6-8c789418cb15" containerName="payload-pre-processing" Oct 29 23:07:33.124825 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.124755 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.151171 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.151137 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 29 23:07:33.151352 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.151184 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bfzvw\"" Oct 29 23:07:33.151352 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.151208 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 29 23:07:33.151352 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.151238 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 29 23:07:33.153598 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.153569 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86789cb564-4vgtn"] Oct 29 23:07:33.164766 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.164740 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-webhook-cert\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.164925 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.164787 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-metrics-certs\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.164925 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.164897 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzpr\" (UniqueName: \"kubernetes.io/projected/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-kube-api-access-hdzpr\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.192141 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.192103 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:33.266244 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.266208 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-webhook-cert\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.266442 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.266267 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-metrics-certs\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.266442 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.266327 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzpr\" (UniqueName: \"kubernetes.io/projected/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-kube-api-access-hdzpr\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.268900 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.268875 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-webhook-cert\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.269033 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.268929 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-metrics-certs\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.284937 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.284910 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzpr\" (UniqueName: \"kubernetes.io/projected/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-kube-api-access-hdzpr\") pod \"maas-controller-86789cb564-4vgtn\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.441683 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.441594 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:33.635150 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.635108 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f71636-1271-4e26-b3a4-cad6448fb72a" path="/var/lib/kubelet/pods/f7f71636-1271-4e26-b3a4-cad6448fb72a/volumes" Oct 29 23:07:33.737970 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.737935 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86789cb564-4vgtn"] Oct 29 23:07:33.741740 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:07:33.741561 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ca15a7_4a8e_4ae0_8a4e_15a237c7ccc1.slice/crio-d522c86250b2be8ce3c7e08617d644d93919cb2343e3faf03ec006b62b322d72 WatchSource:0}: Error finding container d522c86250b2be8ce3c7e08617d644d93919cb2343e3faf03ec006b62b322d72: Status 404 returned error can't find the container with id d522c86250b2be8ce3c7e08617d644d93919cb2343e3faf03ec006b62b322d72 Oct 29 23:07:33.743032 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:33.743013 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:07:34.393206 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:34.393164 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-4vgtn" event={"ID":"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1","Type":"ContainerStarted","Data":"5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1"} Oct 29 23:07:34.393449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:34.393222 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-4vgtn" event={"ID":"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1","Type":"ContainerStarted","Data":"d522c86250b2be8ce3c7e08617d644d93919cb2343e3faf03ec006b62b322d72"} Oct 29 23:07:34.393449 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:34.393260 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:34.433277 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:34.433215 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-86789cb564-4vgtn" podStartSLOduration=0.938887491 podStartE2EDuration="1.433202793s" totalImagesPullingTime="494.315302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:34.432585273 +0000 UTC m=+3116.913902773" watchObservedRunningTime="2026-10-29 23:07:34.433202793 +0000 UTC m=+3116.914520285" Oct 29 23:07:35.488211 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:35.488169 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:38.350345 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:38.350307 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:38.489312 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:38.489253 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:39.011649 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:39.011613 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:41.452291 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:41.452257 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:43.709333 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:07:43.709287 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:07:44.479288 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:44.479254 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:07:45.401675 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:45.401597 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:07:49.312885 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.312843 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn"] Oct 29 23:07:49.313483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.313466 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="29da9dd4-983c-4343-9439-005194023ce1" containerName="istio-proxy" Oct 29 23:07:49.313483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.313486 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="55441a0e-ab81-44df-8301-956c67fecc3b" containerName="istio-proxy" Oct 29 23:07:49.313598 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.313505 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77602a7-cf53-413f-9484-7baccce25ea5" containerName="istio-proxy" Oct 29 23:07:49.313598 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.313516 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce818887-c1a6-452b-9584-432c17e64752" containerName="istio-proxy" Oct 29 23:07:49.318272 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.318244 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.345931 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.345892 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn"] Oct 29 23:07:49.427848 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.427803 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrtd\" (UniqueName: \"kubernetes.io/projected/da6465f3-9c91-4654-9e71-ce4bc457286c-kube-api-access-mbrtd\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.428009 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.427881 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.428009 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.427929 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.428129 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.428016 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.428129 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.428050 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.428129 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.428118 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.428239 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.428138 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.428239 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.428213 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da6465f3-9c91-4654-9e71-ce4bc457286c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.428387 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.428364 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529020 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.528978 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrtd\" (UniqueName: \"kubernetes.io/projected/da6465f3-9c91-4654-9e71-ce4bc457286c-kube-api-access-mbrtd\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529020 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529025 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529051 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529069 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529096 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529165 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529190 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529281 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529217 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da6465f3-9c91-4654-9e71-ce4bc457286c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529299 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529484 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529669 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529605 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.529845 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.529685 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.530084 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.530056 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.530271 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.530249 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da6465f3-9c91-4654-9e71-ce4bc457286c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.531540 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.531515 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.531633 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.531576 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.548021 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.547991 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da6465f3-9c91-4654-9e71-ce4bc457286c-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.548165 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.548083 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrtd\" (UniqueName: \"kubernetes.io/projected/da6465f3-9c91-4654-9e71-ce4bc457286c-kube-api-access-mbrtd\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn\" (UID: \"da6465f3-9c91-4654-9e71-ce4bc457286c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.633780 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.633695 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:49.943543 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.943501 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn"] Oct 29 23:07:49.945741 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:07:49.945713 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6465f3_9c91_4654_9e71_ce4bc457286c.slice/crio-9f9fa40eaefd92a00756283caf0b25ead767a16046d06ba119e43620b608d9d0 WatchSource:0}: Error finding container 9f9fa40eaefd92a00756283caf0b25ead767a16046d06ba119e43620b608d9d0: Status 404 returned error can't find the container with id 9f9fa40eaefd92a00756283caf0b25ead767a16046d06ba119e43620b608d9d0 Oct 29 23:07:49.947905 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.947873 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:07:49.947979 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.947943 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:07:49.947979 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:49.947971 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:07:50.488350 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:50.488311 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" event={"ID":"da6465f3-9c91-4654-9e71-ce4bc457286c","Type":"ContainerStarted","Data":"9c791b1e72438827bcb0b0d51d3ddb99e3051d222e51d119f8eb23194442f804"} Oct 29 23:07:50.488918 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:50.488366 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" event={"ID":"da6465f3-9c91-4654-9e71-ce4bc457286c","Type":"ContainerStarted","Data":"9f9fa40eaefd92a00756283caf0b25ead767a16046d06ba119e43620b608d9d0"} Oct 29 23:07:50.554009 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:50.553931 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podStartSLOduration=1.553913191 podStartE2EDuration="1.553913191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:50.542046252 +0000 UTC m=+3133.023363749" watchObservedRunningTime="2026-10-29 23:07:50.553913191 +0000 UTC m=+3133.035230688" Oct 29 23:07:50.634514 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:50.634478 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:07:50.636469 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:50.636440 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:07:50.636646 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:50.636502 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 29 23:07:51.634098 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:51.634055 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:07:51.634629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:51.634112 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 29 23:07:52.634564 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:52.634524 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:07:52.635008 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:52.634578 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 29 23:07:53.634409 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:53.634367 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:07:53.634577 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:53.634457 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 29 23:07:54.634910 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:54.634874 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:07:54.635350 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:54.634933 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 29 23:07:55.634723 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:55.634692 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:07:55.634915 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:55.634745 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 29 23:07:56.634374 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:56.634336 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 29 23:07:56.634816 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:56.634409 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 29 23:07:58.634856 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:58.634813 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:07:58.635305 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:58.634878 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:07:59.676705 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:59.676654 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:07:59.677148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:07:59.676754 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:00.268275 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:00.268241 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:08:00.268693 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:00.268668 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:08:00.269263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:00.269244 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" Oct 29 23:08:04.334366 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.334289 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:04.355651 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.355604 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj"] Oct 29 23:08:04.359835 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.359811 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.390846 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.390816 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj"] Oct 29 23:08:04.392422 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.392369 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9"] Oct 29 23:08:04.396251 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.396220 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.396601 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.396567 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.396771 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.396632 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba68353b-41a6-4375-bd34-0613a7820cda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.396771 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.396764 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.396895 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.396812 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnhp\" (UniqueName: \"kubernetes.io/projected/ba68353b-41a6-4375-bd34-0613a7820cda-kube-api-access-ghnhp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.396939 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.396916 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba68353b-41a6-4375-bd34-0613a7820cda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.396980 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.396962 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba68353b-41a6-4375-bd34-0613a7820cda-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.397030 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.396989 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.397153 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.397129 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.397225 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.397172 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.420658 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.420616 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9"] Oct 29 23:08:04.497960 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.497919 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.497969 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.497990 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498059 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h67k\" (UniqueName: \"kubernetes.io/projected/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-kube-api-access-2h67k\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498095 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498117 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498144 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba68353b-41a6-4375-bd34-0613a7820cda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498218 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498315 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498447 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498363 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498471 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498501 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498512 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498527 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498552 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.498639 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498559 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498645 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnhp\" (UniqueName: \"kubernetes.io/projected/ba68353b-41a6-4375-bd34-0613a7820cda-kube-api-access-ghnhp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498750 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba68353b-41a6-4375-bd34-0613a7820cda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498795 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498798 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba68353b-41a6-4375-bd34-0613a7820cda-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.498967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.498848 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.499284 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.499192 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.499352 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.499323 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba68353b-41a6-4375-bd34-0613a7820cda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.500553 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.500531 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba68353b-41a6-4375-bd34-0613a7820cda-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.500777 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.500754 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba68353b-41a6-4375-bd34-0613a7820cda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.517685 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.517649 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba68353b-41a6-4375-bd34-0613a7820cda-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.517844 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.517807 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnhp\" (UniqueName: \"kubernetes.io/projected/ba68353b-41a6-4375-bd34-0613a7820cda-kube-api-access-ghnhp\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj\" (UID: \"ba68353b-41a6-4375-bd34-0613a7820cda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.599708 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.599608 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.599708 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.599663 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h67k\" (UniqueName: \"kubernetes.io/projected/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-kube-api-access-2h67k\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.599708 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.599695 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600036 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.599734 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600036 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.599756 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600036 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.599889 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600036 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.600012 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600264 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.600058 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600264 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.600081 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600382 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.600267 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.600416 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.600430 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600604 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.600484 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.600768 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.600746 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.602108 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.602091 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.602443 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.602427 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.618629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.618589 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.618813 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.618793 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h67k\" (UniqueName: \"kubernetes.io/projected/e4c2752b-2cfb-42c2-bc99-a2237cc6dfac-kube-api-access-2h67k\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9\" (UID: \"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.674380 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.674342 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:04.713224 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.713190 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:04.990174 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.990140 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj"] Oct 29 23:08:04.991946 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:08:04.991917 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba68353b_41a6_4375_bd34_0613a7820cda.slice/crio-03f24c7a178f0f862546455103a21136462a3da522b18e30cfd4eeef9d2c5ce3 WatchSource:0}: Error finding container 03f24c7a178f0f862546455103a21136462a3da522b18e30cfd4eeef9d2c5ce3: Status 404 returned error can't find the container with id 03f24c7a178f0f862546455103a21136462a3da522b18e30cfd4eeef9d2c5ce3 Oct 29 23:08:04.994416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.994362 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:04.994490 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.994456 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:04.994538 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:04.994493 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:05.027750 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.027724 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9"] Oct 29 23:08:05.030522 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:08:05.030374 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c2752b_2cfb_42c2_bc99_a2237cc6dfac.slice/crio-26eddeec26b824a09d8abbcdf6b02030cf74c974a30e4a8660aaf1353605f597 WatchSource:0}: Error finding container 26eddeec26b824a09d8abbcdf6b02030cf74c974a30e4a8660aaf1353605f597: Status 404 returned error can't find the container with id 26eddeec26b824a09d8abbcdf6b02030cf74c974a30e4a8660aaf1353605f597 Oct 29 23:08:05.033029 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.032990 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:05.033120 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.033081 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:05.033167 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.033121 5610 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 29 23:08:05.581384 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.581273 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:05.597880 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.597802 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" event={"ID":"ba68353b-41a6-4375-bd34-0613a7820cda","Type":"ContainerStarted","Data":"c980322633b9c1ca0028d7eea8099759621dfbc5134e9689d536cd37c4c4b938"} Oct 29 23:08:05.598276 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.598176 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" event={"ID":"ba68353b-41a6-4375-bd34-0613a7820cda","Type":"ContainerStarted","Data":"03f24c7a178f0f862546455103a21136462a3da522b18e30cfd4eeef9d2c5ce3"} Oct 29 23:08:05.607560 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.607516 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" event={"ID":"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac","Type":"ContainerStarted","Data":"5ae8f0a24506fca08f4ff2860e6aaa856b164466ebacd65be6b9520d634dda20"} Oct 29 23:08:05.607748 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.607606 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" event={"ID":"e4c2752b-2cfb-42c2-bc99-a2237cc6dfac","Type":"ContainerStarted","Data":"26eddeec26b824a09d8abbcdf6b02030cf74c974a30e4a8660aaf1353605f597"} Oct 29 23:08:05.655361 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.655308 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podStartSLOduration=1.655292755 podStartE2EDuration="1.655292755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:05.653109445 +0000 UTC m=+3148.134426939" watchObservedRunningTime="2026-10-29 23:08:05.655292755 +0000 UTC m=+3148.136610245" Oct 29 23:08:05.677415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.670925 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86789cb564-4vgtn"] Oct 29 23:08:05.677415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.671208 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-86789cb564-4vgtn" podUID="a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" containerName="manager" containerID="cri-o://5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1" gracePeriod=30 Oct 29 23:08:05.681978 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.681938 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:05.721776 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.719819 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:05.775416 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:05.775326 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podStartSLOduration=1.775306036 podStartE2EDuration="1.775306036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:05.719594229 +0000 UTC m=+3148.200911735" watchObservedRunningTime="2026-10-29 23:08:05.775306036 +0000 UTC m=+3148.256623530" Oct 29 23:08:06.073421 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.071634 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:08:06.125462 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.125428 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-metrics-certs\") pod \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " Oct 29 23:08:06.125681 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.125496 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-kube-api-access-hdzpr\" (UniqueName: \"kubernetes.io/projected/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-kube-api-access-hdzpr\") pod \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " Oct 29 23:08:06.125681 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.125597 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-webhook-cert\") pod \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\" (UID: \"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1\") " Oct 29 23:08:06.129610 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.128751 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-metrics-certs" pod "a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" (UID: "a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:06.129610 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.128775 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-kube-api-access-hdzpr" pod "a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" (UID: "a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1"). InnerVolumeSpecName "kube-api-access-hdzpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:08:06.129610 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.129556 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-webhook-cert" pod "a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" (UID: "a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:08:06.226906 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.226826 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:08:06.226906 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.226857 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdzpr\" (UniqueName: \"kubernetes.io/projected/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-kube-api-access-hdzpr\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:08:06.226906 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.226868 5610 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1-webhook-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:08:06.616260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.616164 5610 generic.go:356] "Generic (PLEG): container finished" podID="a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" containerID="5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1" exitCode=0 Oct 29 23:08:06.616260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.616214 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-4vgtn" Oct 29 23:08:06.616260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.616243 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-4vgtn" event={"ID":"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1","Type":"ContainerDied","Data":"5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1"} Oct 29 23:08:06.616891 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.616316 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-4vgtn" event={"ID":"a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1","Type":"ContainerDied","Data":"d522c86250b2be8ce3c7e08617d644d93919cb2343e3faf03ec006b62b322d72"} Oct 29 23:08:06.616891 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.616456 5610 scope.go:122] "RemoveContainer" containerID="5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1" Oct 29 23:08:06.627112 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.627091 5610 scope.go:122] "RemoveContainer" containerID="5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1" Oct 29 23:08:06.627574 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:08:06.627546 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1\": container with ID starting with 5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1 not found: ID does not exist" containerID="5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1" Oct 29 23:08:06.627662 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.627584 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1"} err="failed to get container status \"5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1\": rpc error: code = NotFound desc = could not find container \"5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1\": container with ID starting with 5a22b270d842c8df2ecdad2e5dd9a780a1176ee5a62d7b711828795e3df05eb1 not found: ID does not exist" Oct 29 23:08:06.671413 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.665945 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86789cb564-4vgtn"] Oct 29 23:08:06.679418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.677978 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86789cb564-4vgtn"] Oct 29 23:08:06.695419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.680876 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:08:06.695419 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.680931 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 29 23:08:06.715619 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.715577 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 29 23:08:06.715809 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:06.715648 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podUID="e4c2752b-2cfb-42c2-bc99-a2237cc6dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 29 23:08:07.635351 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.635308 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" path="/var/lib/kubelet/pods/a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1/volumes" Oct 29 23:08:07.685576 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.685531 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:08:07.685908 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.685868 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 29 23:08:07.714606 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.714566 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 29 23:08:07.714825 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.714623 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podUID="e4c2752b-2cfb-42c2-bc99-a2237cc6dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 29 23:08:07.810078 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.810040 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:07.939936 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.939842 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86789cb564-nmrx6"] Oct 29 23:08:07.940645 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.940623 5610 cpu_manager.go:414] "containerMap: removing container" podUID="a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" containerName="manager" Oct 29 23:08:07.940776 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.940765 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" containerName="manager" Oct 29 23:08:07.940969 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.940959 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ca15a7-4a8e-4ae0-8a4e-15a237c7ccc1" containerName="manager" Oct 29 23:08:07.945621 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.945599 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:07.979420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.977479 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 29 23:08:08.000575 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.993086 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86789cb564-nmrx6"] Oct 29 23:08:08.000575 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.993330 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bfzvw\"" Oct 29 23:08:08.000575 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.993441 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 29 23:08:08.000575 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:07.993521 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 29 23:08:08.047488 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.047443 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9gs9\" (UniqueName: \"kubernetes.io/projected/7f610e2e-d332-4761-9372-e26cd50183db-kube-api-access-v9gs9\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.047656 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.047500 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f610e2e-d332-4761-9372-e26cd50183db-webhook-cert\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.047729 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.047672 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f610e2e-d332-4761-9372-e26cd50183db-metrics-certs\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.148682 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.148636 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f610e2e-d332-4761-9372-e26cd50183db-metrics-certs\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.148876 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.148702 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9gs9\" (UniqueName: \"kubernetes.io/projected/7f610e2e-d332-4761-9372-e26cd50183db-kube-api-access-v9gs9\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.148876 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.148736 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f610e2e-d332-4761-9372-e26cd50183db-webhook-cert\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.151470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.151444 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f610e2e-d332-4761-9372-e26cd50183db-webhook-cert\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.151796 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.151779 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f610e2e-d332-4761-9372-e26cd50183db-metrics-certs\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.183263 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.183219 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9gs9\" (UniqueName: \"kubernetes.io/projected/7f610e2e-d332-4761-9372-e26cd50183db-kube-api-access-v9gs9\") pod \"maas-controller-86789cb564-nmrx6\" (UID: \"7f610e2e-d332-4761-9372-e26cd50183db\") " pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.258832 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.258801 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:08.635053 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.634945 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86789cb564-nmrx6"] Oct 29 23:08:08.637435 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:08:08.637406 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f610e2e_d332_4761_9372_e26cd50183db.slice/crio-ad5585622b72404c2bd3799f6d3ecd8d82b0a499ccdda716447dc2b00eea6268 WatchSource:0}: Error finding container ad5585622b72404c2bd3799f6d3ecd8d82b0a499ccdda716447dc2b00eea6268: Status 404 returned error can't find the container with id ad5585622b72404c2bd3799f6d3ecd8d82b0a499ccdda716447dc2b00eea6268 Oct 29 23:08:08.675882 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.675841 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:08:08.676187 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.676148 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 29 23:08:08.714588 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.714558 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 29 23:08:08.714827 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:08.714806 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podUID="e4c2752b-2cfb-42c2-bc99-a2237cc6dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 29 23:08:09.634445 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:09.634386 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-nmrx6" event={"ID":"7f610e2e-d332-4761-9372-e26cd50183db","Type":"ContainerStarted","Data":"f665c6b66e9de6d18bf2475a0c2de2323d83a6838df36a2c0ec22610e4640e12"} Oct 29 23:08:09.634646 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:09.634464 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86789cb564-nmrx6" event={"ID":"7f610e2e-d332-4761-9372-e26cd50183db","Type":"ContainerStarted","Data":"ad5585622b72404c2bd3799f6d3ecd8d82b0a499ccdda716447dc2b00eea6268"} Oct 29 23:08:09.634646 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:09.634534 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:09.686417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:09.686282 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:08:09.686417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:09.686338 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 29 23:08:09.714960 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:09.714808 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 29 23:08:09.714960 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:09.714872 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podUID="e4c2752b-2cfb-42c2-bc99-a2237cc6dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 29 23:08:10.688418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:10.679881 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:08:10.688418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:10.679946 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 29 23:08:10.714568 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:10.714525 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 29 23:08:10.714758 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:10.714585 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podUID="e4c2752b-2cfb-42c2-bc99-a2237cc6dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 29 23:08:11.675519 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:11.675473 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 29 23:08:11.675712 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:11.675546 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 29 23:08:11.714515 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:11.714476 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 29 23:08:11.714890 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:11.714541 5610 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podUID="e4c2752b-2cfb-42c2-bc99-a2237cc6dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 29 23:08:13.512802 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:13.512764 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:13.513286 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:13.512852 5610 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:13.513366 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:13.513322 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:13.513523 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:13.513506 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:13.514079 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:13.514057 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" Oct 29 23:08:13.514239 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:13.514227 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" Oct 29 23:08:13.566666 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:13.566606 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-86789cb564-nmrx6" podStartSLOduration=6.086088053 podStartE2EDuration="6.566591593s" totalImagesPullingTime="480.50354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:09.684382142 +0000 UTC m=+3152.165699636" watchObservedRunningTime="2026-10-29 23:08:13.566591593 +0000 UTC m=+3156.047909083" Oct 29 23:08:13.722626 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:08:13.722566 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:08:20.642598 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:20.642513 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86789cb564-nmrx6" Oct 29 23:08:43.476418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:43.458159 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:08:43.949197 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:08:43.949159 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:08:44.520721 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:44.520683 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:08:44.521244 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:44.520763 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:08:47.348484 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:08:47.348444 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:13.709905 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:09:13.709863 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:09:41.694969 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:09:41.694911 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:09:43.708826 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:09:43.708783 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:10:01.295873 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:10:01.295770 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:10:01.296464 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:10:01.295883 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:10:11.129140 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:10:11.129100 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:13.724854 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:10:13.724778 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:10:41.811233 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:10:41.811194 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:10:43.710062 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:10:43.710022 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:11:13.732204 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:11:13.732155 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:11:16.322533 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:11:16.322488 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:16.323159 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:11:16.323126 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:41.950196 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:11:41.949963 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:11:43.823418 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:11:43.823097 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:11:44.540115 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:11:44.539739 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:44.540115 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:11:44.539828 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:11:44.556560 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:11:44.556516 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:11:44.556771 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:11:44.556598 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podUID="e4c2752b-2cfb-42c2-bc99-a2237cc6dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:12:10.121041 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:10.121003 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:13.708828 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:12:13.708788 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:12:23.556311 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:23.556270 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:24.042004 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:24.041948 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:24.191483 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:24.191443 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:27.621094 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:27.621057 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:27.849930 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:27.849894 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:29.018974 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.018925 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:29.169531 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.169495 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:29.313663 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.313583 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:29.421025 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.420990 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7"] Oct 29 23:12:29.425014 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.424992 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:29.445289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.445256 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7"] Oct 29 23:12:29.450450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.450377 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7e3223\"" Oct 29 23:12:29.450450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.450408 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-841d46b3-dockercfg-44wn7\"" Oct 29 23:12:29.501136 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.501088 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:29.512581 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.512521 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t"] Oct 29 23:12:29.518592 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.518562 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:29.554737 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.554691 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7\" (UID: \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:29.555039 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.555010 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bdm\" (UniqueName: \"kubernetes.io/projected/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-kube-api-access-j2bdm\") pod \"payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7\" (UID: \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:29.560477 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.560423 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t"] Oct 29 23:12:29.656425 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.656320 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ac73b-29f4-4348-a270-42524bbbd3ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t\" (UID: \"939ac73b-29f4-4348-a270-42524bbbd3ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:29.656582 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.656458 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7\" (UID: \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:29.656582 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.656488 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bdm\" (UniqueName: \"kubernetes.io/projected/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-kube-api-access-j2bdm\") pod \"payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7\" (UID: \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:29.656582 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.656550 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghdx\" (UniqueName: \"kubernetes.io/projected/939ac73b-29f4-4348-a270-42524bbbd3ef-kube-api-access-dghdx\") pod \"payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t\" (UID: \"939ac73b-29f4-4348-a270-42524bbbd3ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:29.657151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.657129 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7\" (UID: \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:29.677666 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.677630 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bdm\" (UniqueName: \"kubernetes.io/projected/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-kube-api-access-j2bdm\") pod \"payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7\" (UID: \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:29.738026 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.737989 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:29.766151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.758330 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghdx\" (UniqueName: \"kubernetes.io/projected/939ac73b-29f4-4348-a270-42524bbbd3ef-kube-api-access-dghdx\") pod \"payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t\" (UID: \"939ac73b-29f4-4348-a270-42524bbbd3ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:29.766151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.758474 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ac73b-29f4-4348-a270-42524bbbd3ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t\" (UID: \"939ac73b-29f4-4348-a270-42524bbbd3ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:29.766151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.760374 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ac73b-29f4-4348-a270-42524bbbd3ef-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t\" (UID: \"939ac73b-29f4-4348-a270-42524bbbd3ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:29.779932 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.779904 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghdx\" (UniqueName: \"kubernetes.io/projected/939ac73b-29f4-4348-a270-42524bbbd3ef-kube-api-access-dghdx\") pod \"payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t\" (UID: \"939ac73b-29f4-4348-a270-42524bbbd3ef\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:29.872977 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:29.872941 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:30.039650 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.039622 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7"] Oct 29 23:12:30.041948 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:12:30.041915 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d55d76_3637_4a58_8dd7_7eebd0bb9ff0.slice/crio-e1e1fefe05394609ecc437357401c97584b723ca226df625311beaac58362c6c WatchSource:0}: Error finding container e1e1fefe05394609ecc437357401c97584b723ca226df625311beaac58362c6c: Status 404 returned error can't find the container with id e1e1fefe05394609ecc437357401c97584b723ca226df625311beaac58362c6c Oct 29 23:12:30.174574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.174499 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t"] Oct 29 23:12:30.178283 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:12:30.178248 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939ac73b_29f4_4348_a270_42524bbbd3ef.slice/crio-b4dffcf13704430516b0fb5133e458404223fa9fff5871b487eeee18a045cb94 WatchSource:0}: Error finding container b4dffcf13704430516b0fb5133e458404223fa9fff5871b487eeee18a045cb94: Status 404 returned error can't find the container with id b4dffcf13704430516b0fb5133e458404223fa9fff5871b487eeee18a045cb94 Oct 29 23:12:30.308594 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.308553 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" event={"ID":"939ac73b-29f4-4348-a270-42524bbbd3ef","Type":"ContainerStarted","Data":"8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b"} Oct 29 23:12:30.308813 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.308617 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" event={"ID":"939ac73b-29f4-4348-a270-42524bbbd3ef","Type":"ContainerStarted","Data":"b4dffcf13704430516b0fb5133e458404223fa9fff5871b487eeee18a045cb94"} Oct 29 23:12:30.308813 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.308734 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:30.310131 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.310101 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" event={"ID":"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0","Type":"ContainerStarted","Data":"1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e"} Oct 29 23:12:30.310265 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.310155 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" event={"ID":"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0","Type":"ContainerStarted","Data":"e1e1fefe05394609ecc437357401c97584b723ca226df625311beaac58362c6c"} Oct 29 23:12:30.310265 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.310182 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:30.349653 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.349592 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" podStartSLOduration=1.349576657 podStartE2EDuration="1.349576657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:30.348314839 +0000 UTC m=+3412.829632332" watchObservedRunningTime="2026-10-29 23:12:30.349576657 +0000 UTC m=+3412.830894144" Oct 29 23:12:30.388151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.388094 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" podStartSLOduration=1.388080056 podStartE2EDuration="1.388080056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:30.386641256 +0000 UTC m=+3412.867958750" watchObservedRunningTime="2026-10-29 23:12:30.388080056 +0000 UTC m=+3412.869397547" Oct 29 23:12:30.665420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:30.663706 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:32.303808 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:32.303764 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:33.827428 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:33.827375 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:35.423435 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:35.423371 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:36.922001 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:36.921966 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:37.396210 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:37.396159 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:38.077537 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:38.077498 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:38.258248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:38.258212 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:38.716381 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:38.716347 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:40.262564 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:40.262532 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:40.861633 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:40.861602 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:41.318264 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:41.318235 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:12:41.318785 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:41.318307 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:12:41.842986 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:41.842924 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:42.063109 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:42.063070 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:43.513961 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:43.513914 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:43.708954 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:12:43.708916 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:12:45.009266 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:45.009229 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:46.815992 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:46.815951 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:48.374500 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:48.374459 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:49.902738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:49.900330 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:51.315141 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:51.315076 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:52.770472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:52.770418 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:54.236793 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:54.236742 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:55.739923 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:55.739829 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:57.462450 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:57.462369 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:12:58.944766 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:12:58.944732 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:00.528348 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:00.528300 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:02.051914 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:02.049192 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:03.516819 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:03.516730 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:04.942297 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:04.942252 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:06.509627 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:06.509588 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:08.050423 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:08.049687 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:09.499146 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:09.499106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:11.046909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:11.046871 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:12.501200 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:12.501161 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:13.710740 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:13:13.710704 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:13:14.090533 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:14.090285 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:14.323884 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:14.323843 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:15.810813 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:15.810772 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:17.515340 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:17.515301 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:19.005756 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:19.005701 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:19.214572 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:19.214517 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:20.626048 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:20.626004 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:22.129659 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:22.129624 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:23.633206 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:23.632988 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:24.700759 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:24.700708 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t"] Oct 29 23:13:24.738004 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:24.737967 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7"] Oct 29 23:13:24.907067 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:24.907026 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:25.061741 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:25.061649 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:26.135365 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:26.135327 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:26.285175 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:26.285134 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:26.792012 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:26.791978 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:29.701725 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:29.701679 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" podUID="939ac73b-29f4-4348-a270-42524bbbd3ef" containerName="payload-processing" containerID="cri-o://8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b" gracePeriod=25 Oct 29 23:13:29.738703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:29.738636 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" podUID="d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" containerName="payload-pre-processing" containerID="cri-o://1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e" gracePeriod=25 Oct 29 23:13:31.315939 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:31.315897 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" podUID="939ac73b-29f4-4348-a270-42524bbbd3ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:13:31.316420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:31.315903 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" podUID="d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:13:32.477705 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:32.477622 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:32.639044 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:32.639002 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:32.875909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:32.875765 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:41.316473 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:41.316433 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" podUID="939ac73b-29f4-4348-a270-42524bbbd3ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:13:41.316993 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:41.316434 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" podUID="d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:13:42.160596 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:42.160557 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:43.709988 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:13:43.709953 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:13:49.993809 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:49.993781 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:13:50.018594 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.018567 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:13:50.116818 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.116785 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939ac73b-29f4-4348-a270-42524bbbd3ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ac73b-29f4-4348-a270-42524bbbd3ef-plugins-config-volume\") pod \"939ac73b-29f4-4348-a270-42524bbbd3ef\" (UID: \"939ac73b-29f4-4348-a270-42524bbbd3ef\") " Oct 29 23:13:50.117013 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.116904 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939ac73b-29f4-4348-a270-42524bbbd3ef-kube-api-access-dghdx\" (UniqueName: \"kubernetes.io/projected/939ac73b-29f4-4348-a270-42524bbbd3ef-kube-api-access-dghdx\") pod \"939ac73b-29f4-4348-a270-42524bbbd3ef\" (UID: \"939ac73b-29f4-4348-a270-42524bbbd3ef\") " Oct 29 23:13:50.117013 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.117004 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-kube-api-access-j2bdm\" (UniqueName: \"kubernetes.io/projected/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-kube-api-access-j2bdm\") pod \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\" (UID: \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\") " Oct 29 23:13:50.117148 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.117058 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-plugins-config-volume\") pod \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\" (UID: \"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0\") " Oct 29 23:13:50.117233 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.117211 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939ac73b-29f4-4348-a270-42524bbbd3ef-plugins-config-volume" pod "939ac73b-29f4-4348-a270-42524bbbd3ef" (UID: "939ac73b-29f4-4348-a270-42524bbbd3ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:50.117602 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.117573 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-plugins-config-volume" pod "d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" (UID: "d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:13:50.117694 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.117616 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ac73b-29f4-4348-a270-42524bbbd3ef-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:13:50.119094 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.119067 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939ac73b-29f4-4348-a270-42524bbbd3ef-kube-api-access-dghdx" pod "939ac73b-29f4-4348-a270-42524bbbd3ef" (UID: "939ac73b-29f4-4348-a270-42524bbbd3ef"). InnerVolumeSpecName "kube-api-access-dghdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:50.119165 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.119108 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-kube-api-access-j2bdm" pod "d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" (UID: "d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0"). InnerVolumeSpecName "kube-api-access-j2bdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:50.218506 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.218465 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:13:50.218506 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.218500 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghdx\" (UniqueName: \"kubernetes.io/projected/939ac73b-29f4-4348-a270-42524bbbd3ef-kube-api-access-dghdx\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:13:50.218506 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.218510 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bdm\" (UniqueName: \"kubernetes.io/projected/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0-kube-api-access-j2bdm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:13:50.705980 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.705941 5610 generic.go:356] "Generic (PLEG): container finished" podID="939ac73b-29f4-4348-a270-42524bbbd3ef" containerID="8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b" exitCode=0 Oct 29 23:13:50.706178 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.705987 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" Oct 29 23:13:50.706178 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.706031 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" event={"ID":"939ac73b-29f4-4348-a270-42524bbbd3ef","Type":"ContainerDied","Data":"8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b"} Oct 29 23:13:50.706178 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.706085 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t" event={"ID":"939ac73b-29f4-4348-a270-42524bbbd3ef","Type":"ContainerDied","Data":"b4dffcf13704430516b0fb5133e458404223fa9fff5871b487eeee18a045cb94"} Oct 29 23:13:50.706178 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.706116 5610 scope.go:122] "RemoveContainer" containerID="8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b" Oct 29 23:13:50.707511 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.707488 5610 generic.go:356] "Generic (PLEG): container finished" podID="d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" containerID="1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e" exitCode=0 Oct 29 23:13:50.707621 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.707529 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" Oct 29 23:13:50.707621 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.707562 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" event={"ID":"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0","Type":"ContainerDied","Data":"1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e"} Oct 29 23:13:50.707621 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.707604 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7" event={"ID":"d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0","Type":"ContainerDied","Data":"e1e1fefe05394609ecc437357401c97584b723ca226df625311beaac58362c6c"} Oct 29 23:13:50.726112 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.726093 5610 scope.go:122] "RemoveContainer" containerID="8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b" Oct 29 23:13:50.726457 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:13:50.726437 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b\": container with ID starting with 8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b not found: ID does not exist" containerID="8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b" Oct 29 23:13:50.726551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.726464 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b"} err="failed to get container status \"8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b\": rpc error: code = NotFound desc = could not find container \"8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b\": container with ID starting with 8ac7d48581027427985938c91d93d21374944807b7c6dfa04df49f76fc66b56b not found: ID does not exist" Oct 29 23:13:50.726551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.726485 5610 scope.go:122] "RemoveContainer" containerID="1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e" Oct 29 23:13:50.734280 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.734259 5610 scope.go:122] "RemoveContainer" containerID="1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e" Oct 29 23:13:50.734588 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:13:50.734568 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e\": container with ID starting with 1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e not found: ID does not exist" containerID="1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e" Oct 29 23:13:50.734667 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.734595 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e"} err="failed to get container status \"1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e\": rpc error: code = NotFound desc = could not find container \"1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e\": container with ID starting with 1f22d92c8f9100f1d4b6089ebd01cc3f9f65b241b9dd714f21a1475e9f58ac4e not found: ID does not exist" Oct 29 23:13:50.769740 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.769698 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7"] Oct 29 23:13:50.781170 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.781136 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7e3223-dbdf5nv4j7"] Oct 29 23:13:50.807780 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.807749 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t"] Oct 29 23:13:50.812049 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:50.812018 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7e3223-7bdb8948859k9t"] Oct 29 23:13:51.634443 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:51.634410 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939ac73b-29f4-4348-a270-42524bbbd3ef" path="/var/lib/kubelet/pods/939ac73b-29f4-4348-a270-42524bbbd3ef/volumes" Oct 29 23:13:51.634846 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:51.634790 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" path="/var/lib/kubelet/pods/d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0/volumes" Oct 29 23:13:51.969267 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:51.969225 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw"] Oct 29 23:13:51.969491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:51.969462 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" podUID="70fdbd02-4506-4534-a9b0-95ebee3b33b5" containerName="manager" containerID="cri-o://e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e" gracePeriod=10 Oct 29 23:13:52.230615 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.230543 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 23:13:52.337613 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.337573 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fdbd02-4506-4534-a9b0-95ebee3b33b5-kube-api-access-bv5xj\" (UniqueName: \"kubernetes.io/projected/70fdbd02-4506-4534-a9b0-95ebee3b33b5-kube-api-access-bv5xj\") pod \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\" (UID: \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\") " Oct 29 23:13:52.337819 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.337696 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fdbd02-4506-4534-a9b0-95ebee3b33b5-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70fdbd02-4506-4534-a9b0-95ebee3b33b5-extensions-socket-volume\") pod \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\" (UID: \"70fdbd02-4506-4534-a9b0-95ebee3b33b5\") " Oct 29 23:13:52.338159 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.338132 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fdbd02-4506-4534-a9b0-95ebee3b33b5-extensions-socket-volume" pod "70fdbd02-4506-4534-a9b0-95ebee3b33b5" (UID: "70fdbd02-4506-4534-a9b0-95ebee3b33b5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 29 23:13:52.339951 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.339926 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fdbd02-4506-4534-a9b0-95ebee3b33b5-kube-api-access-bv5xj" pod "70fdbd02-4506-4534-a9b0-95ebee3b33b5" (UID: "70fdbd02-4506-4534-a9b0-95ebee3b33b5"). InnerVolumeSpecName "kube-api-access-bv5xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:13:52.439546 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.439510 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv5xj\" (UniqueName: \"kubernetes.io/projected/70fdbd02-4506-4534-a9b0-95ebee3b33b5-kube-api-access-bv5xj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:13:52.439546 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.439542 5610 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70fdbd02-4506-4534-a9b0-95ebee3b33b5-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:13:52.728069 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.728032 5610 generic.go:356] "Generic (PLEG): container finished" podID="70fdbd02-4506-4534-a9b0-95ebee3b33b5" containerID="e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e" exitCode=0 Oct 29 23:13:52.728603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.728078 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" event={"ID":"70fdbd02-4506-4534-a9b0-95ebee3b33b5","Type":"ContainerDied","Data":"e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e"} Oct 29 23:13:52.728603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.728082 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" Oct 29 23:13:52.728603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.728118 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw" event={"ID":"70fdbd02-4506-4534-a9b0-95ebee3b33b5","Type":"ContainerDied","Data":"5ad7dfa7130e68e7c51ef50fa65c72b1b96de9471179283100c221651250968e"} Oct 29 23:13:52.728603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.728139 5610 scope.go:122] "RemoveContainer" containerID="e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e" Oct 29 23:13:52.737313 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.737293 5610 scope.go:122] "RemoveContainer" containerID="e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e" Oct 29 23:13:52.737605 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:13:52.737583 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e\": container with ID starting with e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e not found: ID does not exist" containerID="e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e" Oct 29 23:13:52.737685 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.737613 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e"} err="failed to get container status \"e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e\": rpc error: code = NotFound desc = could not find container \"e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e\": container with ID starting with e5ef350ca169769243c964f18050330d8852c5dba7ed37a55a4a55da8f7e029e not found: ID does not exist" Oct 29 23:13:52.798803 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.798764 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw"] Oct 29 23:13:52.806538 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:52.806503 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-5shkw"] Oct 29 23:13:53.634293 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:53.634258 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70fdbd02-4506-4534-a9b0-95ebee3b33b5" path="/var/lib/kubelet/pods/70fdbd02-4506-4534-a9b0-95ebee3b33b5/volumes" Oct 29 23:13:55.997325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:55.997286 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:13:56.885591 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:13:56.885559 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:02.587517 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:02.587478 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:13.241760 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:13.241727 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:13.392563 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:13.392532 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:13.708142 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:14:13.708105 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:14:42.286748 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:42.286715 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:14:43.709022 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:14:43.708980 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:14:55.139289 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139254 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk"] Oct 29 23:14:55.139761 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139746 5610 cpu_manager.go:414] "containerMap: removing container" podUID="d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" containerName="payload-pre-processing" Oct 29 23:14:55.139808 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139763 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" containerName="payload-pre-processing" Oct 29 23:14:55.139808 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139777 5610 cpu_manager.go:414] "containerMap: removing container" podUID="939ac73b-29f4-4348-a270-42524bbbd3ef" containerName="payload-processing" Oct 29 23:14:55.139808 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139783 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939ac73b-29f4-4348-a270-42524bbbd3ef" containerName="payload-processing" Oct 29 23:14:55.139808 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139805 5610 cpu_manager.go:414] "containerMap: removing container" podUID="70fdbd02-4506-4534-a9b0-95ebee3b33b5" containerName="manager" Oct 29 23:14:55.139949 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139810 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fdbd02-4506-4534-a9b0-95ebee3b33b5" containerName="manager" Oct 29 23:14:55.139949 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139871 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fdbd02-4506-4534-a9b0-95ebee3b33b5" containerName="manager" Oct 29 23:14:55.139949 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139882 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d55d76-3637-4a58-8dd7-7eebd0bb9ff0" containerName="payload-pre-processing" Oct 29 23:14:55.139949 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.139889 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="939ac73b-29f4-4348-a270-42524bbbd3ef" containerName="payload-processing" Oct 29 23:14:55.144773 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.144756 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:55.159675 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.159649 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk"] Oct 29 23:14:55.169982 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.169944 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qjqmw\"" Oct 29 23:14:55.252990 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.252949 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2ws\" (UniqueName: \"kubernetes.io/projected/cdc2914f-173e-4e48-ba29-860ff8b60bb3-kube-api-access-9p2ws\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j27wk\" (UID: \"cdc2914f-173e-4e48-ba29-860ff8b60bb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:55.253219 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.253001 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdc2914f-173e-4e48-ba29-860ff8b60bb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j27wk\" (UID: \"cdc2914f-173e-4e48-ba29-860ff8b60bb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:55.353962 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.353921 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2ws\" (UniqueName: \"kubernetes.io/projected/cdc2914f-173e-4e48-ba29-860ff8b60bb3-kube-api-access-9p2ws\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j27wk\" (UID: \"cdc2914f-173e-4e48-ba29-860ff8b60bb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:55.354174 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.353978 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdc2914f-173e-4e48-ba29-860ff8b60bb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j27wk\" (UID: \"cdc2914f-173e-4e48-ba29-860ff8b60bb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:55.354475 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.354454 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cdc2914f-173e-4e48-ba29-860ff8b60bb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j27wk\" (UID: \"cdc2914f-173e-4e48-ba29-860ff8b60bb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:55.381999 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.381957 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2ws\" (UniqueName: \"kubernetes.io/projected/cdc2914f-173e-4e48-ba29-860ff8b60bb3-kube-api-access-9p2ws\") pod \"kuadrant-operator-controller-manager-57bb9bfff-j27wk\" (UID: \"cdc2914f-173e-4e48-ba29-860ff8b60bb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:55.457930 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.457889 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:55.776906 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:55.776819 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk"] Oct 29 23:14:55.791265 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:14:55.791230 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc2914f_173e_4e48_ba29_860ff8b60bb3.slice/crio-84c4a0cc5bb53f4a853b4fcc0287eb4ab9e16c9a7f596a4f792d53c38ce3a4c7 WatchSource:0}: Error finding container 84c4a0cc5bb53f4a853b4fcc0287eb4ab9e16c9a7f596a4f792d53c38ce3a4c7: Status 404 returned error can't find the container with id 84c4a0cc5bb53f4a853b4fcc0287eb4ab9e16c9a7f596a4f792d53c38ce3a4c7 Oct 29 23:14:56.032366 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:56.032270 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" event={"ID":"cdc2914f-173e-4e48-ba29-860ff8b60bb3","Type":"ContainerStarted","Data":"870ab1ec00cedc5e2dab23de6e7ce51142f237993e509ac6a9209cc4772acbaf"} Oct 29 23:14:56.032366 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:56.032342 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:14:56.032582 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:56.032370 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" event={"ID":"cdc2914f-173e-4e48-ba29-860ff8b60bb3","Type":"ContainerStarted","Data":"84c4a0cc5bb53f4a853b4fcc0287eb4ab9e16c9a7f596a4f792d53c38ce3a4c7"} Oct 29 23:14:56.074516 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:14:56.074455 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" podStartSLOduration=1.074440503 podStartE2EDuration="1.074440503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:56.073994185 +0000 UTC m=+3558.555311678" watchObservedRunningTime="2026-10-29 23:14:56.074440503 +0000 UTC m=+3558.555757996" Oct 29 23:15:02.046266 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:02.046227 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:07.039996 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:07.039963 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-j27wk" Oct 29 23:15:13.710449 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:15:13.710412 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:15:19.736846 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:19.736812 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:27.898854 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:27.898816 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:28.031026 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:28.030991 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:42.412553 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:42.412513 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:43.710596 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:15:43.710549 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:15:48.225907 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:48.225870 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:48.745215 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:48.745171 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:48.924083 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:48.924040 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:51.908781 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:51.908695 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:52.081898 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:52.081855 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:53.443000 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.442961 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:53.616857 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.616822 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:53.679248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.679212 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5"] Oct 29 23:15:53.683241 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.683218 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:53.700774 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.700714 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2250ca\"" Oct 29 23:15:53.713409 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.713371 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5"] Oct 29 23:15:53.713617 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.713598 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-52f4dd28-dockercfg-s7jnn\"" Oct 29 23:15:53.788574 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.788533 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:53.790152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.790113 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj"] Oct 29 23:15:53.795667 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.795643 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:53.817062 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.817031 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj"] Oct 29 23:15:53.856420 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.856373 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7bc\" (UniqueName: \"kubernetes.io/projected/8238ca93-d96d-48c1-92a6-b217e57e1cc7-kube-api-access-5q7bc\") pod \"payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5\" (UID: \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:53.856592 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.856435 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8238ca93-d96d-48c1-92a6-b217e57e1cc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5\" (UID: \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:53.957798 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.957704 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42e507a-dfa7-4eee-b40b-8db20ed55896-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj\" (UID: \"a42e507a-dfa7-4eee-b40b-8db20ed55896\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:53.957798 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.957775 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7bc\" (UniqueName: \"kubernetes.io/projected/8238ca93-d96d-48c1-92a6-b217e57e1cc7-kube-api-access-5q7bc\") pod \"payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5\" (UID: \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:53.958007 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.957859 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8238ca93-d96d-48c1-92a6-b217e57e1cc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5\" (UID: \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:53.958007 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.957901 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68682\" (UniqueName: \"kubernetes.io/projected/a42e507a-dfa7-4eee-b40b-8db20ed55896-kube-api-access-68682\") pod \"payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj\" (UID: \"a42e507a-dfa7-4eee-b40b-8db20ed55896\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:53.958678 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.958660 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8238ca93-d96d-48c1-92a6-b217e57e1cc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5\" (UID: \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:53.976072 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.976046 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7bc\" (UniqueName: \"kubernetes.io/projected/8238ca93-d96d-48c1-92a6-b217e57e1cc7-kube-api-access-5q7bc\") pod \"payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5\" (UID: \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:53.995890 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:53.995856 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:54.059486 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.059219 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68682\" (UniqueName: \"kubernetes.io/projected/a42e507a-dfa7-4eee-b40b-8db20ed55896-kube-api-access-68682\") pod \"payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj\" (UID: \"a42e507a-dfa7-4eee-b40b-8db20ed55896\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:54.059486 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.059384 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42e507a-dfa7-4eee-b40b-8db20ed55896-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj\" (UID: \"a42e507a-dfa7-4eee-b40b-8db20ed55896\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:54.060533 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.060499 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42e507a-dfa7-4eee-b40b-8db20ed55896-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj\" (UID: \"a42e507a-dfa7-4eee-b40b-8db20ed55896\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:54.079150 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.079114 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68682\" (UniqueName: \"kubernetes.io/projected/a42e507a-dfa7-4eee-b40b-8db20ed55896-kube-api-access-68682\") pod \"payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj\" (UID: \"a42e507a-dfa7-4eee-b40b-8db20ed55896\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:54.109967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.109938 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:54.290997 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.290500 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5"] Oct 29 23:15:54.293272 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:15:54.293243 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8238ca93_d96d_48c1_92a6_b217e57e1cc7.slice/crio-a834a229d8ce3865a19a3b6fc2c2b94ad1c5cf7e874f873844f89193fb1a0c0f WatchSource:0}: Error finding container a834a229d8ce3865a19a3b6fc2c2b94ad1c5cf7e874f873844f89193fb1a0c0f: Status 404 returned error can't find the container with id a834a229d8ce3865a19a3b6fc2c2b94ad1c5cf7e874f873844f89193fb1a0c0f Oct 29 23:15:54.390736 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.390702 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" event={"ID":"8238ca93-d96d-48c1-92a6-b217e57e1cc7","Type":"ContainerStarted","Data":"902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273"} Oct 29 23:15:54.390916 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.390758 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" event={"ID":"8238ca93-d96d-48c1-92a6-b217e57e1cc7","Type":"ContainerStarted","Data":"a834a229d8ce3865a19a3b6fc2c2b94ad1c5cf7e874f873844f89193fb1a0c0f"} Oct 29 23:15:54.390916 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.390787 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:15:54.411200 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.411172 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj"] Oct 29 23:15:54.414350 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:15:54.414321 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42e507a_dfa7_4eee_b40b_8db20ed55896.slice/crio-68e742a729d1a634e4e8ffb12b8f189f50cdc60d251472eacee0c85eae7355fe WatchSource:0}: Error finding container 68e742a729d1a634e4e8ffb12b8f189f50cdc60d251472eacee0c85eae7355fe: Status 404 returned error can't find the container with id 68e742a729d1a634e4e8ffb12b8f189f50cdc60d251472eacee0c85eae7355fe Oct 29 23:15:54.438470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:54.438412 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" podStartSLOduration=1.438383355 podStartE2EDuration="1.438383355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:54.436880518 +0000 UTC m=+3616.918198023" watchObservedRunningTime="2026-10-29 23:15:54.438383355 +0000 UTC m=+3616.919700847" Oct 29 23:15:55.132603 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:55.132564 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:55.397023 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:55.396931 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" event={"ID":"a42e507a-dfa7-4eee-b40b-8db20ed55896","Type":"ContainerStarted","Data":"f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab"} Oct 29 23:15:55.397023 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:55.397016 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" event={"ID":"a42e507a-dfa7-4eee-b40b-8db20ed55896","Type":"ContainerStarted","Data":"68e742a729d1a634e4e8ffb12b8f189f50cdc60d251472eacee0c85eae7355fe"} Oct 29 23:15:55.397278 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:55.397107 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:15:55.437845 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:55.437784 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" podStartSLOduration=2.437769687 podStartE2EDuration="2.437769687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:15:55.436774003 +0000 UTC m=+3617.918091510" watchObservedRunningTime="2026-10-29 23:15:55.437769687 +0000 UTC m=+3617.919087182" Oct 29 23:15:56.670703 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:56.670625 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:58.281854 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:58.281813 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:15:59.763221 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:15:59.763186 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:01.381763 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:01.381670 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:02.852135 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:02.852091 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:04.499182 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:04.499148 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:05.400299 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:05.400261 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:16:05.851293 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:05.851234 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:06.404607 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:06.404573 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:16:06.823738 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:06.823699 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:07.489331 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:07.489294 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:07.646070 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:07.646034 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:07.799952 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:07.799867 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:09.148198 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:09.148161 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:10.320460 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:10.320422 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:10.530716 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:10.530649 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:12.248792 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:12.248755 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:13.711073 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:16:13.711025 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:16:13.750048 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:13.749923 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:15.335493 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:15.335456 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:16.846719 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:16.846683 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:18.292261 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:18.292222 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:19.853107 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:19.853069 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:21.251427 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:21.248633 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:22.721785 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:22.721744 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:24.160097 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:24.160057 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:25.928426 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:25.928058 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:27.434041 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:27.434000 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:29.130154 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:29.130106 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:30.658353 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:30.656473 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:32.190895 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:32.190858 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:33.844326 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:33.844246 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:35.378617 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:35.378579 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:36.945256 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:36.944822 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:37.301517 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:37.301425 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:38.807851 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:38.807805 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:40.253727 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:40.253569 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:42.123472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:42.123431 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:42.488278 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:42.488245 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:42.740538 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:42.740459 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:43.708964 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:16:43.708925 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:16:44.041361 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:44.041272 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:45.657355 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:45.657319 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:47.358472 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:47.358438 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:47.938238 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:47.938206 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj"] Oct 29 23:16:47.976150 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:47.976113 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5"] Oct 29 23:16:48.182994 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:48.182946 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:48.499552 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:48.499511 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:49.246308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:49.246267 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:50.748064 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:50.748028 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:51.373325 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:51.373286 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:51.699337 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:51.699304 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:52.219735 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:52.219705 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:52.939306 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:52.939261 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" podUID="a42e507a-dfa7-4eee-b40b-8db20ed55896" containerName="payload-processing" containerID="cri-o://f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab" gracePeriod=25 Oct 29 23:16:52.977152 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:52.977105 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" podUID="8238ca93-d96d-48c1-92a6-b217e57e1cc7" containerName="payload-pre-processing" containerID="cri-o://902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273" gracePeriod=25 Oct 29 23:16:55.398605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:55.398566 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" podUID="8238ca93-d96d-48c1-92a6-b217e57e1cc7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:16:56.403195 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:56.403150 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" podUID="a42e507a-dfa7-4eee-b40b-8db20ed55896" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:16:59.650634 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:59.650584 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:16:59.953764 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:16:59.953731 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:00.163309 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:00.163271 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:05.264551 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:05.264512 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:05.398909 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:05.398869 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" podUID="8238ca93-d96d-48c1-92a6-b217e57e1cc7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:17:06.402573 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:06.402529 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" podUID="a42e507a-dfa7-4eee-b40b-8db20ed55896" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 29 23:17:10.158418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:10.156520 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:10.158418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:10.156597 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:17:13.224172 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.224141 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:17:13.246867 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.246840 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:17:13.272670 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.272643 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42e507a-dfa7-4eee-b40b-8db20ed55896-kube-api-access-68682\" (UniqueName: \"kubernetes.io/projected/a42e507a-dfa7-4eee-b40b-8db20ed55896-kube-api-access-68682\") pod \"a42e507a-dfa7-4eee-b40b-8db20ed55896\" (UID: \"a42e507a-dfa7-4eee-b40b-8db20ed55896\") " Oct 29 23:17:13.272829 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.272808 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42e507a-dfa7-4eee-b40b-8db20ed55896-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42e507a-dfa7-4eee-b40b-8db20ed55896-plugins-config-volume\") pod \"a42e507a-dfa7-4eee-b40b-8db20ed55896\" (UID: \"a42e507a-dfa7-4eee-b40b-8db20ed55896\") " Oct 29 23:17:13.273232 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.273189 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42e507a-dfa7-4eee-b40b-8db20ed55896-plugins-config-volume" pod "a42e507a-dfa7-4eee-b40b-8db20ed55896" (UID: "a42e507a-dfa7-4eee-b40b-8db20ed55896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:17:13.274796 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.274775 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42e507a-dfa7-4eee-b40b-8db20ed55896-kube-api-access-68682" pod "a42e507a-dfa7-4eee-b40b-8db20ed55896" (UID: "a42e507a-dfa7-4eee-b40b-8db20ed55896"). InnerVolumeSpecName "kube-api-access-68682". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:13.373942 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.373908 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8238ca93-d96d-48c1-92a6-b217e57e1cc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8238ca93-d96d-48c1-92a6-b217e57e1cc7-plugins-config-volume\") pod \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\" (UID: \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\") " Oct 29 23:17:13.374133 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.374016 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8238ca93-d96d-48c1-92a6-b217e57e1cc7-kube-api-access-5q7bc\" (UniqueName: \"kubernetes.io/projected/8238ca93-d96d-48c1-92a6-b217e57e1cc7-kube-api-access-5q7bc\") pod \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\" (UID: \"8238ca93-d96d-48c1-92a6-b217e57e1cc7\") " Oct 29 23:17:13.374293 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.374268 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8238ca93-d96d-48c1-92a6-b217e57e1cc7-plugins-config-volume" pod "8238ca93-d96d-48c1-92a6-b217e57e1cc7" (UID: "8238ca93-d96d-48c1-92a6-b217e57e1cc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:17:13.374370 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.374280 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42e507a-dfa7-4eee-b40b-8db20ed55896-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:17:13.374370 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.374316 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68682\" (UniqueName: \"kubernetes.io/projected/a42e507a-dfa7-4eee-b40b-8db20ed55896-kube-api-access-68682\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:17:13.376089 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.376066 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8238ca93-d96d-48c1-92a6-b217e57e1cc7-kube-api-access-5q7bc" pod "8238ca93-d96d-48c1-92a6-b217e57e1cc7" (UID: "8238ca93-d96d-48c1-92a6-b217e57e1cc7"). InnerVolumeSpecName "kube-api-access-5q7bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:17:13.475207 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.475117 5610 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8238ca93-d96d-48c1-92a6-b217e57e1cc7-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:17:13.475207 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.475148 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7bc\" (UniqueName: \"kubernetes.io/projected/8238ca93-d96d-48c1-92a6-b217e57e1cc7-kube-api-access-5q7bc\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:17:13.709184 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:17:13.709148 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:17:13.846695 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.846607 5610 generic.go:356] "Generic (PLEG): container finished" podID="a42e507a-dfa7-4eee-b40b-8db20ed55896" containerID="f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab" exitCode=0 Oct 29 23:17:13.846695 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.846649 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" Oct 29 23:17:13.846924 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.846699 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" event={"ID":"a42e507a-dfa7-4eee-b40b-8db20ed55896","Type":"ContainerDied","Data":"f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab"} Oct 29 23:17:13.846924 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.846754 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj" event={"ID":"a42e507a-dfa7-4eee-b40b-8db20ed55896","Type":"ContainerDied","Data":"68e742a729d1a634e4e8ffb12b8f189f50cdc60d251472eacee0c85eae7355fe"} Oct 29 23:17:13.846924 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.846793 5610 scope.go:122] "RemoveContainer" containerID="f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab" Oct 29 23:17:13.848043 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.848019 5610 generic.go:356] "Generic (PLEG): container finished" podID="8238ca93-d96d-48c1-92a6-b217e57e1cc7" containerID="902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273" exitCode=0 Oct 29 23:17:13.848136 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.848073 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" Oct 29 23:17:13.848136 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.848097 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" event={"ID":"8238ca93-d96d-48c1-92a6-b217e57e1cc7","Type":"ContainerDied","Data":"902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273"} Oct 29 23:17:13.848136 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.848132 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5" event={"ID":"8238ca93-d96d-48c1-92a6-b217e57e1cc7","Type":"ContainerDied","Data":"a834a229d8ce3865a19a3b6fc2c2b94ad1c5cf7e874f873844f89193fb1a0c0f"} Oct 29 23:17:13.855519 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.855503 5610 scope.go:122] "RemoveContainer" containerID="f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab" Oct 29 23:17:13.855776 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:17:13.855758 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab\": container with ID starting with f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab not found: ID does not exist" containerID="f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab" Oct 29 23:17:13.855833 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.855784 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab"} err="failed to get container status \"f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab\": rpc error: code = NotFound desc = could not find container \"f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab\": container with ID starting with f08b8f30c946568a196f472a65bfa6512c33e031de70905344a12a5218dfc4ab not found: ID does not exist" Oct 29 23:17:13.855833 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.855800 5610 scope.go:122] "RemoveContainer" containerID="902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273" Oct 29 23:17:13.863510 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.863489 5610 scope.go:122] "RemoveContainer" containerID="902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273" Oct 29 23:17:13.863782 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:17:13.863757 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273\": container with ID starting with 902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273 not found: ID does not exist" containerID="902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273" Oct 29 23:17:13.863843 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.863792 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273"} err="failed to get container status \"902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273\": rpc error: code = NotFound desc = could not find container \"902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273\": container with ID starting with 902e3a29ecd01dd8dd90cbdc0d80f48d48973761cd99e82f6a14c8d0e9fce273 not found: ID does not exist" Oct 29 23:17:13.894879 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.894849 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj"] Oct 29 23:17:13.906463 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.906436 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2250ca-54857dfd6zt4fj"] Oct 29 23:17:13.930795 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.930766 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5"] Oct 29 23:17:13.934491 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:13.934466 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2250ca-6bbd98qzw5"] Oct 29 23:17:15.634005 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:15.633967 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8238ca93-d96d-48c1-92a6-b217e57e1cc7" path="/var/lib/kubelet/pods/8238ca93-d96d-48c1-92a6-b217e57e1cc7/volumes" Oct 29 23:17:15.634518 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:15.634501 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42e507a-dfa7-4eee-b40b-8db20ed55896" path="/var/lib/kubelet/pods/a42e507a-dfa7-4eee-b40b-8db20ed55896/volumes" Oct 29 23:17:19.041813 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:19.039894 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:20.065819 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:20.065729 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:25.485361 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:25.485319 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:27.984593 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:27.984558 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:42.678451 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:42.678384 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:43.708881 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:17:43.708836 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:17:46.340681 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:46.340156 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:46.340681 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:46.340228 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn" podUID="da6465f3-9c91-4654-9e71-ce4bc457286c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:17:54.586431 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:54.585215 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:56.820569 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:56.820536 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:17:59.569417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:59.568532 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:59.569417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:59.568643 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj" podUID="ba68353b-41a6-4375-bd34-0613a7820cda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:17:59.569417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:59.569293 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:17:59.569417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:17:59.569355 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9" podUID="e4c2752b-2cfb-42c2-bc99-a2237cc6dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:18:13.869121 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:18:13.869029 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:18:42.710093 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.710054 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-slbq6"] Oct 29 23:18:42.710660 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.710503 5610 cpu_manager.go:414] "containerMap: removing container" podUID="a42e507a-dfa7-4eee-b40b-8db20ed55896" containerName="payload-processing" Oct 29 23:18:42.710660 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.710515 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42e507a-dfa7-4eee-b40b-8db20ed55896" containerName="payload-processing" Oct 29 23:18:42.710660 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.710534 5610 cpu_manager.go:414] "containerMap: removing container" podUID="8238ca93-d96d-48c1-92a6-b217e57e1cc7" containerName="payload-pre-processing" Oct 29 23:18:42.710660 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.710540 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8238ca93-d96d-48c1-92a6-b217e57e1cc7" containerName="payload-pre-processing" Oct 29 23:18:42.710660 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.710608 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="8238ca93-d96d-48c1-92a6-b217e57e1cc7" containerName="payload-pre-processing" Oct 29 23:18:42.710660 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.710623 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42e507a-dfa7-4eee-b40b-8db20ed55896" containerName="payload-processing" Oct 29 23:18:42.713971 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.713948 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.736102 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.736078 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-slbq6"] Oct 29 23:18:42.770234 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.770198 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602f326d-0027-45c0-90ac-e696ec394ef9-openshift-service-ca\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.770428 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.770252 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602f326d-0027-45c0-90ac-e696ec394ef9-maas-api-tls\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.770428 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.770383 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bg29\" (UniqueName: \"kubernetes.io/projected/602f326d-0027-45c0-90ac-e696ec394ef9-kube-api-access-5bg29\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.770538 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.770522 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602f326d-0027-45c0-90ac-e696ec394ef9-metrics-certs\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.857967 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.857919 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:18:42.871061 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.871028 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602f326d-0027-45c0-90ac-e696ec394ef9-metrics-certs\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.871248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.871074 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602f326d-0027-45c0-90ac-e696ec394ef9-openshift-service-ca\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.871248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.871177 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602f326d-0027-45c0-90ac-e696ec394ef9-maas-api-tls\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.871248 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.871234 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bg29\" (UniqueName: \"kubernetes.io/projected/602f326d-0027-45c0-90ac-e696ec394ef9-kube-api-access-5bg29\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.872034 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.872011 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602f326d-0027-45c0-90ac-e696ec394ef9-openshift-service-ca\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.873810 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.873783 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602f326d-0027-45c0-90ac-e696ec394ef9-maas-api-tls\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.873912 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.873833 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602f326d-0027-45c0-90ac-e696ec394ef9-metrics-certs\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:42.890729 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:42.890699 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bg29\" (UniqueName: \"kubernetes.io/projected/602f326d-0027-45c0-90ac-e696ec394ef9-kube-api-access-5bg29\") pod \"maas-api-595cc648c-slbq6\" (UID: \"602f326d-0027-45c0-90ac-e696ec394ef9\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:43.027646 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:43.027566 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:43.322282 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:43.322183 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-slbq6"] Oct 29 23:18:43.325379 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:18:43.325348 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602f326d_0027_45c0_90ac_e696ec394ef9.slice/crio-3264384cc3c0a3b72595f8c64e8d9ef231bd9f0f447c21679f0e3e311241a559 WatchSource:0}: Error finding container 3264384cc3c0a3b72595f8c64e8d9ef231bd9f0f447c21679f0e3e311241a559: Status 404 returned error can't find the container with id 3264384cc3c0a3b72595f8c64e8d9ef231bd9f0f447c21679f0e3e311241a559 Oct 29 23:18:43.326674 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:43.326657 5610 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 29 23:18:43.500076 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:43.500040 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" event={"ID":"602f326d-0027-45c0-90ac-e696ec394ef9","Type":"ContainerStarted","Data":"3264384cc3c0a3b72595f8c64e8d9ef231bd9f0f447c21679f0e3e311241a559"} Oct 29 23:18:43.714431 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:18:43.714367 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:18:44.509003 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:44.508946 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" event={"ID":"602f326d-0027-45c0-90ac-e696ec394ef9","Type":"ContainerStarted","Data":"fd3604744c425ae2ee7621446e35708bb7b37f04a7d3dc1edf7d10e31d0518e1"} Oct 29 23:18:44.509242 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:44.509052 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:44.552950 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:44.552875 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" podStartSLOduration=2.136353326 podStartE2EDuration="2.552855278s" totalImagesPullingTime="416.501952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:44.551205395 +0000 UTC m=+3787.032522889" watchObservedRunningTime="2026-10-29 23:18:44.552855278 +0000 UTC m=+3787.034172772" Oct 29 23:18:50.527074 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:50.526985 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-slbq6" Oct 29 23:18:50.604066 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:50.604031 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vchhh"] Oct 29 23:18:50.604290 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:50.604264 5610 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" podUID="ebe896cf-231e-4c94-b414-ae90fb96c5dc" containerName="maas-api" containerID="cri-o://8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5" gracePeriod=30 Oct 29 23:18:51.852650 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.852624 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 23:18:51.931172 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.931082 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe896cf-231e-4c94-b414-ae90fb96c5dc-kube-api-access-pndzf\" (UniqueName: \"kubernetes.io/projected/ebe896cf-231e-4c94-b414-ae90fb96c5dc-kube-api-access-pndzf\") pod \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " Oct 29 23:18:51.931172 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.931166 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-metrics-certs\") pod \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " Oct 29 23:18:51.931436 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.931227 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-maas-api-tls\") pod \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " Oct 29 23:18:51.931436 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.931259 5610 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe896cf-231e-4c94-b414-ae90fb96c5dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe896cf-231e-4c94-b414-ae90fb96c5dc-openshift-service-ca\") pod \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\" (UID: \"ebe896cf-231e-4c94-b414-ae90fb96c5dc\") " Oct 29 23:18:51.931788 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.931751 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe896cf-231e-4c94-b414-ae90fb96c5dc-openshift-service-ca" pod "ebe896cf-231e-4c94-b414-ae90fb96c5dc" (UID: "ebe896cf-231e-4c94-b414-ae90fb96c5dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 29 23:18:51.933439 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.933408 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-maas-api-tls" pod "ebe896cf-231e-4c94-b414-ae90fb96c5dc" (UID: "ebe896cf-231e-4c94-b414-ae90fb96c5dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:51.933543 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.933415 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-metrics-certs" pod "ebe896cf-231e-4c94-b414-ae90fb96c5dc" (UID: "ebe896cf-231e-4c94-b414-ae90fb96c5dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 29 23:18:51.933543 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:51.933489 5610 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe896cf-231e-4c94-b414-ae90fb96c5dc-kube-api-access-pndzf" pod "ebe896cf-231e-4c94-b414-ae90fb96c5dc" (UID: "ebe896cf-231e-4c94-b414-ae90fb96c5dc"). InnerVolumeSpecName "kube-api-access-pndzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 29 23:18:52.032260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.032218 5610 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pndzf\" (UniqueName: \"kubernetes.io/projected/ebe896cf-231e-4c94-b414-ae90fb96c5dc-kube-api-access-pndzf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.032260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.032256 5610 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.032260 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.032265 5610 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebe896cf-231e-4c94-b414-ae90fb96c5dc-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.032548 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.032274 5610 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe896cf-231e-4c94-b414-ae90fb96c5dc-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\"" Oct 29 23:18:52.594629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.594424 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" event={"ID":"ebe896cf-231e-4c94-b414-ae90fb96c5dc","Type":"ContainerDied","Data":"8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5"} Oct 29 23:18:52.594629 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.594499 5610 scope.go:122] "RemoveContainer" containerID="8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5" Oct 29 23:18:52.595016 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.594984 5610 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" Oct 29 23:18:52.609743 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.594355 5610 generic.go:356] "Generic (PLEG): container finished" podID="ebe896cf-231e-4c94-b414-ae90fb96c5dc" containerID="8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5" exitCode=0 Oct 29 23:18:52.609743 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.608668 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vchhh" event={"ID":"ebe896cf-231e-4c94-b414-ae90fb96c5dc","Type":"ContainerDied","Data":"301485068e0f2bf34815f894979033ef6edc01b4d6e2599fffcf28791021a1bd"} Oct 29 23:18:52.628072 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.628049 5610 scope.go:122] "RemoveContainer" containerID="8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5" Oct 29 23:18:52.628477 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:18:52.628386 5610 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5\": container with ID starting with 8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5 not found: ID does not exist" containerID="8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5" Oct 29 23:18:52.628586 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.628489 5610 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5"} err="failed to get container status \"8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5\": rpc error: code = NotFound desc = could not find container \"8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5\": container with ID starting with 8935d9df0de1b8633e1e09c34c9ba672ce64a7414c55c9ddd862504a027cb8c5 not found: ID does not exist" Oct 29 23:18:52.661204 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.661173 5610 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vchhh"] Oct 29 23:18:52.672119 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:52.672090 5610 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vchhh"] Oct 29 23:18:53.645417 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:53.644991 5610 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe896cf-231e-4c94-b414-ae90fb96c5dc" path="/var/lib/kubelet/pods/ebe896cf-231e-4c94-b414-ae90fb96c5dc/volumes" Oct 29 23:18:55.151533 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:55.151491 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:18:55.152211 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:18:55.152166 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:19:10.150589 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:10.150529 5610 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 29 23:19:10.151804 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:10.151271 5610 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j" podUID="a1d48ad6-b38a-4ed8-b13a-44b591cd02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 29 23:19:13.708301 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:19:13.708262 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:19:42.915671 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:42.915634 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:43.709185 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:19:43.709150 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:19:44.947577 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:44.947545 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86789cb564-nmrx6_7f610e2e-d332-4761-9372-e26cd50183db/manager/0.log" Oct 29 23:19:46.035700 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:46.035667 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-nk5rv_9de87ff9-d183-4132-b1b4-2542a370c998/manager/0.log" Oct 29 23:19:46.808257 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:46.808225 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-vzk8m_9b8cb14f-a7e7-438d-bd86-de6e21363fb4/copy-manifests/0.log" Oct 29 23:19:46.836775 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:46.836745 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-vzk8m_9b8cb14f-a7e7-438d-bd86-de6e21363fb4/manager/0.log" Oct 29 23:19:47.026893 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:47.026859 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rdht5_a4f450ff-4b49-4058-b373-d66951b19315/proxy/0.log" Oct 29 23:19:48.023470 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:48.023441 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-slbq6_602f326d-0027-45c0-90ac-e696ec394ef9/maas-api/0.log" Oct 29 23:19:48.380624 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:48.380533 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:19:49.905918 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:49.905886 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-n6z8r_e63c0524-70e2-4969-a8bb-2ae37ceaf9b6/postgres/0.log" Oct 29 23:19:51.795602 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:51.795573 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-9g5gh_8154e098-0d55-4597-81fe-74d4dae02b3d/manager/0.log" Oct 29 23:19:51.999756 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:51.999726 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6p7js_1f14e09a-672a-4d00-b240-c940f41939a6/manager/0.log" Oct 29 23:19:52.402024 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:52.401990 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-j27wk_cdc2914f-173e-4e48-ba29-860ff8b60bb3/manager/0.log" Oct 29 23:19:52.865471 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:52.865439 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-dtrnm_eb2cab26-c089-4dfb-b58f-b778174e62a8/manager/0.log" Oct 29 23:19:53.862605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:53.862575 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kg9ds_d31e7e56-219b-4c05-8868-77037656ae70/discovery/0.log" Oct 29 23:19:54.070921 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:54.070885 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b649d648f-h99f8_4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb/kube-auth-proxy/0.log" Oct 29 23:19:54.874986 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:54.874952 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn_da6465f3-9c91-4654-9e71-ce4bc457286c/istio-proxy/0.log" Oct 29 23:19:55.079057 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:55.079024 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j_a1d48ad6-b38a-4ed8-b13a-44b591cd02d5/istio-proxy/0.log" Oct 29 23:19:55.879525 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:55.879494 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9_e4c2752b-2cfb-42c2-bc99-a2237cc6dfac/istio-proxy/0.log" Oct 29 23:19:56.081823 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:56.081789 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj_ba68353b-41a6-4375-bd34-0613a7820cda/istio-proxy/0.log" Oct 29 23:19:56.670153 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:56.670126 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:19:58.207642 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:58.207608 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dwclk_f7c7e838-5a9d-477c-994e-e02cff87ba8c/main/0.log" Oct 29 23:19:58.398911 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:58.398883 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2_04592e8c-6304-4b3e-8f94-2629a2ff3162/main/0.log" Oct 29 23:19:58.584467 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:19:58.584377 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j_10fcae09-de89-49fc-98f1-3bdaf39751b3/main/0.log" Oct 29 23:20:09.652017 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:09.651985 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86789cb564-nmrx6_7f610e2e-d332-4761-9372-e26cd50183db/manager/0.log" Oct 29 23:20:10.301346 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:10.301312 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-nk5rv_9de87ff9-d183-4132-b1b4-2542a370c998/manager/0.log" Oct 29 23:20:11.068298 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:11.068268 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-vzk8m_9b8cb14f-a7e7-438d-bd86-de6e21363fb4/copy-manifests/0.log" Oct 29 23:20:11.098605 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:11.098578 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b95c7d58c-vzk8m_9b8cb14f-a7e7-438d-bd86-de6e21363fb4/manager/0.log" Oct 29 23:20:11.293679 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:11.293651 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-rdht5_a4f450ff-4b49-4058-b373-d66951b19315/proxy/0.log" Oct 29 23:20:12.021197 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:12.021172 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-slbq6_602f326d-0027-45c0-90ac-e696ec394ef9/maas-api/0.log" Oct 29 23:20:13.708557 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:20:13.708522 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:20:13.930454 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:13.930428 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-n6z8r_e63c0524-70e2-4969-a8bb-2ae37ceaf9b6/postgres/0.log" Oct 29 23:20:15.293560 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:15.293529 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-9g5gh_8154e098-0d55-4597-81fe-74d4dae02b3d/manager/0.log" Oct 29 23:20:15.487177 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:15.487149 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6p7js_1f14e09a-672a-4d00-b240-c940f41939a6/manager/0.log" Oct 29 23:20:15.873479 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:15.873446 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-j27wk_cdc2914f-173e-4e48-ba29-860ff8b60bb3/manager/0.log" Oct 29 23:20:16.316298 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:16.316270 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-dtrnm_eb2cab26-c089-4dfb-b58f-b778174e62a8/manager/0.log" Oct 29 23:20:17.050882 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:17.050856 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kg9ds_d31e7e56-219b-4c05-8868-77037656ae70/discovery/0.log" Oct 29 23:20:17.264039 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:17.264010 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b649d648f-h99f8_4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb/kube-auth-proxy/0.log" Oct 29 23:20:18.033991 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:18.033962 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn_da6465f3-9c91-4654-9e71-ce4bc457286c/istio-proxy/0.log" Oct 29 23:20:18.235846 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:18.235817 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j_a1d48ad6-b38a-4ed8-b13a-44b591cd02d5/istio-proxy/0.log" Oct 29 23:20:19.026968 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:19.026939 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9_e4c2752b-2cfb-42c2-bc99-a2237cc6dfac/istio-proxy/0.log" Oct 29 23:20:19.229401 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:19.229352 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj_ba68353b-41a6-4375-bd34-0613a7820cda/istio-proxy/0.log" Oct 29 23:20:19.819122 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:19.819052 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:20:21.139889 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:21.139860 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dwclk_f7c7e838-5a9d-477c-994e-e02cff87ba8c/main/0.log" Oct 29 23:20:21.335698 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:21.335664 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8xtnm2_04592e8c-6304-4b3e-8f94-2629a2ff3162/main/0.log" Oct 29 23:20:21.516383 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:21.516348 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-r2w9j_10fcae09-de89-49fc-98f1-3bdaf39751b3/main/0.log" Oct 29 23:20:43.061697 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:43.061658 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:43.709446 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:20:43.709408 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:20:53.183851 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:53.183815 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:53.324328 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:53.324293 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:20:55.459930 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:55.459892 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86789cb564-nmrx6_7f610e2e-d332-4761-9372-e26cd50183db/manager/0.log" Oct 29 23:20:55.935377 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:55.935299 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86789cb564-nmrx6_7f610e2e-d332-4761-9372-e26cd50183db/manager/0.log" Oct 29 23:20:56.876941 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:56.876913 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:20:57.068895 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:20:57.068863 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:21:13.709040 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:21:13.708990 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:21:39.777198 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:39.777164 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86789cb564-nmrx6_7f610e2e-d332-4761-9372-e26cd50183db/manager/0.log" Oct 29 23:21:40.190589 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:40.190506 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86789cb564-nmrx6_7f610e2e-d332-4761-9372-e26cd50183db/manager/0.log" Oct 29 23:21:41.023277 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:41.023251 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:21:41.200628 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:41.200603 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:21:41.559785 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:41.559750 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:41.720838 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:41.720804 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:43.151445 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:43.151385 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:43.708801 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:21:43.708762 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor" Oct 29 23:21:45.988646 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:45.988611 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5trph_ab618391-4607-4f00-ab02-819dc9897da1/global-pull-secret-syncer/0.log" Oct 29 23:21:46.876456 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:46.876425 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" Oct 29 23:21:47.323418 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:47.323367 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-251.ec2.internal_df1fead76acc43615f3bffca1a53f41e/haproxy/0.log" Oct 29 23:21:53.303198 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:53.303116 5610 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6v7fk_aa39a765-7c7f-4212-a8e2-c7f2a91f7339/konnectivity-agent/0.log" limit=1024 Oct 29 23:21:54.358306 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:54.358275 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-9g5gh_8154e098-0d55-4597-81fe-74d4dae02b3d/manager/0.log" Oct 29 23:21:54.477121 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:54.477089 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-6p7js_1f14e09a-672a-4d00-b240-c940f41939a6/manager/0.log" Oct 29 23:21:54.748377 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:54.748349 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-j27wk_cdc2914f-173e-4e48-ba29-860ff8b60bb3/manager/0.log" Oct 29 23:21:55.018432 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:55.018343 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-dtrnm_eb2cab26-c089-4dfb-b58f-b778174e62a8/manager/0.log" Oct 29 23:21:56.248184 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.248151 5610 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm"] Oct 29 23:21:56.249092 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.249064 5610 cpu_manager.go:414] "containerMap: removing container" podUID="ebe896cf-231e-4c94-b414-ae90fb96c5dc" containerName="maas-api" Oct 29 23:21:56.249092 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.249092 5610 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe896cf-231e-4c94-b414-ae90fb96c5dc" containerName="maas-api" Oct 29 23:21:56.254415 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.251755 5610 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe896cf-231e-4c94-b414-ae90fb96c5dc" containerName="maas-api" Oct 29 23:21:56.256264 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.256225 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.270821 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.270795 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm"] Oct 29 23:21:56.279137 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.279110 5610 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4b5wq\"/\"default-dockercfg-7vvsf\"" Oct 29 23:21:56.279137 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.279118 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4b5wq\"/\"openshift-service-ca.crt\"" Oct 29 23:21:56.279375 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.279150 5610 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4b5wq\"/\"kube-root-ca.crt\"" Oct 29 23:21:56.331308 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.331275 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-podres\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.331524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.331328 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-lib-modules\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.331524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.331440 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-proc\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.331524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.331465 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-sys\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.331524 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.331495 5610 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5r6\" (UniqueName: \"kubernetes.io/projected/58218ebe-016d-47bf-8a7e-591d53b7c46e-kube-api-access-4z5r6\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432220 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432182 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-sys\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432231 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5r6\" (UniqueName: \"kubernetes.io/projected/58218ebe-016d-47bf-8a7e-591d53b7c46e-kube-api-access-4z5r6\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432274 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-podres\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432306 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-lib-modules\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432315 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-sys\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432364 5610 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-proc\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432448 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432410 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-podres\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432697 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432454 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-proc\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.432697 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.432455 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58218ebe-016d-47bf-8a7e-591d53b7c46e-lib-modules\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.449465 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.449443 5610 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5r6\" (UniqueName: \"kubernetes.io/projected/58218ebe-016d-47bf-8a7e-591d53b7c46e-kube-api-access-4z5r6\") pod \"perf-node-gather-daemonset-vrpnm\" (UID: \"58218ebe-016d-47bf-8a7e-591d53b7c46e\") " pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.571001 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.570911 5610 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:56.864888 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:56.864854 5610 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm"] Oct 29 23:21:56.867456 ip-10-0-133-251 kubenswrapper[5610]: W1029 23:21:56.867425 5610 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod58218ebe_016d_47bf_8a7e_591d53b7c46e.slice/crio-f9b325721453cb3d21f15519d05613fa6ec18e8398261d5113268e6947ba070d WatchSource:0}: Error finding container f9b325721453cb3d21f15519d05613fa6ec18e8398261d5113268e6947ba070d: Status 404 returned error can't find the container with id f9b325721453cb3d21f15519d05613fa6ec18e8398261d5113268e6947ba070d Oct 29 23:21:57.403177 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.403151 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5thtk_71813e11-80fa-4c04-ba56-cd1bc57d3f35/kube-state-metrics/0.log" Oct 29 23:21:57.507734 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.507702 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" event={"ID":"58218ebe-016d-47bf-8a7e-591d53b7c46e","Type":"ContainerStarted","Data":"16a68fa1e24b043ef174c9a0d6e2ac957e1f63481658cfbbfebcc2b1d13060f6"} Oct 29 23:21:57.507734 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.507750 5610 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" event={"ID":"58218ebe-016d-47bf-8a7e-591d53b7c46e","Type":"ContainerStarted","Data":"f9b325721453cb3d21f15519d05613fa6ec18e8398261d5113268e6947ba070d"} Oct 29 23:21:57.508007 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.507844 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:21:57.514244 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.514222 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5thtk_71813e11-80fa-4c04-ba56-cd1bc57d3f35/kube-rbac-proxy-main/0.log" Oct 29 23:21:57.541642 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.541595 5610 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" podStartSLOduration=1.54158328 podStartE2EDuration="1.54158328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:21:57.540285205 +0000 UTC m=+3980.021602697" watchObservedRunningTime="2026-10-29 23:21:57.54158328 +0000 UTC m=+3980.022900773" Oct 29 23:21:57.628311 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.628279 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5thtk_71813e11-80fa-4c04-ba56-cd1bc57d3f35/kube-rbac-proxy-self/0.log" Oct 29 23:21:57.862582 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.862552 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-vn7xs_d0c9a67d-21dc-44cf-ab1f-1ef20fba70af/monitoring-plugin/0.log" Oct 29 23:21:57.982919 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:57.982889 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-424dq_3f6e07a6-8957-44f5-9203-425259d84060/node-exporter/0.log" Oct 29 23:21:58.096800 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:58.096764 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-424dq_3f6e07a6-8957-44f5-9203-425259d84060/kube-rbac-proxy/0.log" Oct 29 23:21:58.210122 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:58.210097 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-424dq_3f6e07a6-8957-44f5-9203-425259d84060/init-textfile/0.log" Oct 29 23:21:59.039464 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.039437 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xc2mc_9c71f14d-b3bf-4942-beaa-6cdf18e331b9/kube-rbac-proxy-main/0.log" Oct 29 23:21:59.154593 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.154557 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xc2mc_9c71f14d-b3bf-4942-beaa-6cdf18e331b9/kube-rbac-proxy-self/0.log" Oct 29 23:21:59.268564 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.268532 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xc2mc_9c71f14d-b3bf-4942-beaa-6cdf18e331b9/openshift-state-metrics/0.log" Oct 29 23:21:59.393878 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.393805 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a4657df-fb28-427e-bcd1-b4d0ba583369/prometheus/0.log" Oct 29 23:21:59.508404 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.508375 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a4657df-fb28-427e-bcd1-b4d0ba583369/config-reloader/0.log" Oct 29 23:21:59.623068 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.623041 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a4657df-fb28-427e-bcd1-b4d0ba583369/thanos-sidecar/0.log" Oct 29 23:21:59.737697 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.737664 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a4657df-fb28-427e-bcd1-b4d0ba583369/kube-rbac-proxy-web/0.log" Oct 29 23:21:59.856526 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.856494 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a4657df-fb28-427e-bcd1-b4d0ba583369/kube-rbac-proxy/0.log" Oct 29 23:21:59.971357 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:21:59.971322 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a4657df-fb28-427e-bcd1-b4d0ba583369/kube-rbac-proxy-thanos/0.log" Oct 29 23:22:00.086961 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:00.086860 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4a4657df-fb28-427e-bcd1-b4d0ba583369/init-config-reloader/0.log" Oct 29 23:22:00.915845 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:00.915815 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cccd68d-hlbqr_82b16e7a-7a16-409f-9411-d23156de1fe1/thanos-query/0.log" Oct 29 23:22:01.028316 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:01.028285 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cccd68d-hlbqr_82b16e7a-7a16-409f-9411-d23156de1fe1/kube-rbac-proxy-web/0.log" Oct 29 23:22:01.142130 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:01.142102 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cccd68d-hlbqr_82b16e7a-7a16-409f-9411-d23156de1fe1/kube-rbac-proxy/0.log" Oct 29 23:22:01.256712 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:01.256677 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cccd68d-hlbqr_82b16e7a-7a16-409f-9411-d23156de1fe1/prom-label-proxy/0.log" Oct 29 23:22:01.372262 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:01.372218 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cccd68d-hlbqr_82b16e7a-7a16-409f-9411-d23156de1fe1/kube-rbac-proxy-rules/0.log" Oct 29 23:22:01.486330 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:01.486300 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cccd68d-hlbqr_82b16e7a-7a16-409f-9411-d23156de1fe1/kube-rbac-proxy-metrics/0.log" Oct 29 23:22:03.522697 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:03.522658 5610 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4b5wq/perf-node-gather-daemonset-vrpnm" Oct 29 23:22:08.633320 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:08.633290 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6sljl_0f585004-5478-4841-af5c-81c1c4bd9072/dns/0.log" Oct 29 23:22:08.747599 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:08.747568 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6sljl_0f585004-5478-4841-af5c-81c1c4bd9072/kube-rbac-proxy/0.log" Oct 29 23:22:09.472371 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:09.472339 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lmmsc_d4bb246b-2e00-4a19-a53f-ffcc9d8c0c79/dns-node-resolver/0.log" Oct 29 23:22:11.053836 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:11.053805 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vhtpp_3d4b66ac-6aa8-4ed0-b1d4-0c9b7fc36d09/node-ca/0.log" Oct 29 23:22:11.233606 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:11.233563 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-kg9ds_d31e7e56-219b-4c05-8868-77037656ae70/discovery/0.log" Oct 29 23:22:11.264726 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:11.264693 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b649d648f-h99f8_4e22eb3f-2e6a-46e1-9873-fa5f386cc6fb/kube-auth-proxy/0.log" Oct 29 23:22:11.519201 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:11.519169 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dmvpn_da6465f3-9c91-4654-9e71-ce4bc457286c/istio-proxy/0.log" Oct 29 23:22:11.579054 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:11.579022 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-g2j2j_a1d48ad6-b38a-4ed8-b13a-44b591cd02d5/istio-proxy/0.log" Oct 29 23:22:11.821807 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:11.821735 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-s6jn9_e4c2752b-2cfb-42c2-bc99-a2237cc6dfac/istio-proxy/0.log" Oct 29 23:22:11.882151 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:11.882123 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-tw8xj_ba68353b-41a6-4375-bd34-0613a7820cda/istio-proxy/0.log" Oct 29 23:22:12.060047 ip-10-0-133-251 kubenswrapper[5610]: I1029 23:22:12.060017 5610 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-zt6f2_0554501a-13e6-4813-b5d0-b5cb2b14f4c5/payload-processing/0.log" Oct 29 23:22:13.714540 ip-10-0-133-251 kubenswrapper[5610]: E1029 23:22:13.714503 5610 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-446fn" containerName="extractor"